-
Notifications
You must be signed in to change notification settings - Fork 34
Description
Describe the bug
I am working on the mirroring trafic use case for modelmesh. I have deployed the traffic controller v0.17.1 as there is no mirroring usecase in the iter8 docs for later versions.
The controller resources get created, the pod spins up, but I get error logs relating to the metrics and A/B?n service and BadgerDB:
time=2023-09-22 12:03:02 level=error msg=cannot read config file: /config/metrics.yaml stack-trace=below ...
::Trace:: open /config/metrics.yaml: no such file or directory
time=2023-09-22 12:03:02 level=error msg=unable to read metrics configuration: cannot read config file: /config/metrics.yaml
time=2023-09-22 12:03:02 level=error msg=count not start A/B/n metrics service
time=2023-09-22 12:03:02 level=error msg=Unable to configure metrics storage client cannot open BadgerDB
time=2023-09-22 12:03:02 level=error msg=cound not start A/B/n service
This doesn't kill the pod, its status is "Running".
When I deploy a candidate model and use the mirroring traffic template, the virtual service remains intact, it does not get altered based on the weights configmaps.
To Reproduce
the controller was deployed using this kustomization file:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- 'https://github.com/iter8-tools/iter8.git/kustomize/controller/clusterScoped?ref=v0.17.1'
namespace: iter8
patches:
- patch: |-
- op: replace
path: /data/config.yaml
value: |
clusterScoped: true
defaultResync: 15m
image: iter8/iter8:0.17
logLevel: debug
resourceTypes:
cm:
Group: ""
Resource: configmaps
Version: v1
deploy:
Group: apps
Resource: deployments
Version: v1
conditions:
- name: Available
status: "True"
isvc:
Group: serving.kserve.io
Resource: inferenceservices
Version: v1beta1
conditions:
- name: Ready
status: "True"
svc:
Group: ""
Resource: services
Version: v1
vs:
Group: networking.istio.io
Resource: virtualservices
Version: v1alpha3
storageClassName: ""
target:
kind: ConfigMap
name: iter8
- patch: |-
- op: replace
path: /spec/storageClassName
value: default
target:
kind: PersistentVolumeClaim
name: iter8
Expected behavior
When I deploy a candidate model and the mirroring traffic templates, I would expect traffic to be mirrored to the second model. This does not happen as the virtual service does not get updated with the weights.
Desktop (please complete the following information):
- OS: MacOS but running in kubernetes
- Output of the
iter8 versioncommand: irrelevant as using controller
Additional context
Add any other context about the problem here.