add otel sidecar to llamastack deployment

Signed-off-by: sallyom <somalley@redhat.com>

Author: sallyom

kubernetes/llama-stack/deployment.yaml

@@ -8,8 +8,8 @@ spec:
       app: llamastack
   template:
     metadata:
-      #annotations:
-      #  sidecar.opentelemetry.io/inject: otelsidecar
+      annotations:
+        sidecar.opentelemetry.io/inject: llamastack-otelsidecar
       labels:
         app: llamastack
     spec:
@@ -47,7 +47,7 @@ spec:
             - name: OTEL_SERVICE_NAME
               value: om-llamastack
             - name: OTEL_TRACE_ENDPOINT
-              value: 'http://otel-collector-collector.observability-hub.svc.cluster.local:4318/v1/traces'
+              value: 'http://localhost:4318/v1/traces'
             - name: SAFETY_MODEL
               value: meta-llama/Llama-Guard-3-8B
             - name: SAFETY_VLLM_URL

kubernetes/llama-stack/otel-collector-sidecar.yaml

@@ -0,0 +1,58 @@
+# Once this exists, any pod with the template.metadata.annotation below will send metrics
+# to observability-hub:
+# sidecar.opentelemetry.io/inject: llamastack-otelsidecar
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  name: llamastack-otelsidecar
+spec:
+  observability:
+    metrics: {}
+  deploymentUpdateStrategy: {}
+  config:
+    exporters:
+      debug: {}
+      otlphttp:
+        # all sidecars export to the central observability-hub otel-collector, then be
+        # exported to various backends from there (in-cluster, external 3rd party)
+        # this is deployed with ../observability/otel-collector manifests
+        # see ../observability/README.md for how to deploy this collector
+        endpoint: 'http://otel-collector-collector.observability-hub.svc.cluster.local:4318'
+        tls:
+          insecure: true
+    processors: {}
+    receivers:
+      otlp:
+        protocols:
+          grpc: {}
+          http: {}
+    service:
+      pipelines:
+        traces:
+          exporters:
+            - debug
+            - otlphttp
+          receivers:
+            - otlp
+      telemetry:
+        metrics:
+          address: '0.0.0.0:8888'
+  mode: sidecar
+  resources: {}
+  podDnsConfig: {}
+  managementState: managed
+  upgradeStrategy: automatic
+  ingress:
+    route: {}
+  daemonSetUpdateStrategy: {}
+  targetAllocator:
+    allocationStrategy: consistent-hashing
+    filterStrategy: relabel-config
+    observability:
+      metrics: {}
+    prometheusCR:
+      scrapeInterval: 30s
+    resources: {}
+  replicas: 1
+  ipFamilyPolicy: SingleStack
+

kubernetes/observability/README.md

@@ -65,34 +65,47 @@ However, if exporting off-cluster to a 3rd party observability vendor, the colle
 and can provide a single place with which to receive telemetry from various workloads and export as a single authenticated and
 secure OTLP stream.
 
+#### Central OpenTelemetry Collector
+
 To create a central opentelemetry-collector, update the
 [otel-collector/otel-collector.yaml](./otel-collector/otel-collector.yaml) to match your requirements and then apply.
 
 ```bash
 oc apply --kustomize ./otel-collector -n observability-hub
 ```
 
-### OpenTelemetryCollector Sidecars deployment
+#### OpenTelemetryCollector Sidecars deployment
 
 You can add individual metrics endpoints to the central otel-collector in observability-hub, but
 another way is to add otel-collector sidecar containers to individual deployments throughout the
 cluster. Paired with an annotation on the deployment, telemetry will be exported as configured.
-Any deployment with the annotation below will receive and export telemetry as configured in the
+
+Any deployment with the template.metadata.annotations `sidecar.opentelemetry.io/inject: vllm-otelsidecar`
+will receive and export telemetry as configured in the
 [otel-collector-vllm-sidecar.yaml](./otel-collector/otel-collector-vllm-sidecar.yaml).
 
-The example here will add an otel-collector sidecar custom resource to the `llama-serve` namespace,
-and to trigger a sidecar container, annotate any deployment's `template.metadata.annotations` with:
-`sidecar.opentelemetry.io/inject: vllm-otelsidecar`
+Any deployment with the template.metadata.annotations `sidecar.opentelemetry.io/inject: llamastack-otelsidecar`
+will receive and export telemetry as configured in the
+[otel-collector-llamstack-sidecar.yaml](./otel-collector/otel-collector-llamastack-sidecar.yaml).
+
+The example below will add otel-collector sidecar custom resources to the `llama-serve` namespace,
+and upon a scale down, scale up of the deployments with the added annotations, sidecar otel-collector
+containers will be added to the pods.
 
 ```bash
-oc apply -f ./otel-collector/otel-collector-vllm-sidecar.yaml
+oc apply -f ./otel-collector/otel-collector-vllm-sidecar.yaml -n llama-serve
+oc apply -f ./otel-collector/otel-collector-llamastack-sidecar.yaml -n llama-serve
+
+# Then, annotate whatever deployment you'd like to collect telemetry from
+# Add the annotation to the deployment's `template.metadata.annotations` from the console.
+# OR
+# Patch or modify the llamastack and vLLM deployments with the appropriate annotation.
+# Replace `deployment-name`, `namespace`, and `name-of-otelsideccar` in the below command.
 
-# Then, annotate whatever vllm deployment you'd like to collect metrics from
-# Or, add the annotation to the deployment's `template.metadata.annotations` from the console.
-oc patch deployment <deployment-name> \
-  -n <namespace> \
+oc patch deployment deployment-name \
+  -n namespace \
   --type='merge' \
-  -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.opentelemetry.io/inject":"vllm-otelsidecar"}}}}}'
+  -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.opentelemetry.io/inject":"name-of-otelsidecar"}}}}}'
 ```
 
 ### Cluster Observability Operator Tracing UIPlugin