add otel configs in llamastack manifests

Signed-off-by: sallyom <somalley@redhat.com>

Author: sallyom

kubernetes/observability/README.md

@@ -2,6 +2,53 @@
 
 Follow this README to configure an observability stack in OpenShift to visualize Llamastack telemetry and vLLM metrics.
 
+## Generate telemetry from Llamastack and vLLM
+
+### vLLM
+
+For vLLM, metrics are generated by default and are exposed at `vllm-endpoint:port/metrics`. For a list of metrics,
+you can `curl localhost:8000/metrics` from within a vLLM container.
+
+### Llamastack
+
+With Llamastack, you need to specify in the run-config.yaml to enable telemetry collection with an opentelemetry receiver.
+Here's how to do that:
+
+#### Updated manifests for telemetry trace collection with opentelemetry receiver endpoint
+
+This is for traces only. There is a similar `otel_metric` sink and `otel_metric_endpoint`, however, there are currently
+only 4 metrics generated within Llamastack, and these are duplicates of what vLLM provides.
+
+[kubernetes/llama-stack/configmap.yaml](../llama-stack/configmap.yaml)
+
+```yaml
+---
+      telemetry:
+      - provider_id: meta-reference
+        provider_type: inline::meta-reference
+        config:
+          service_name: ${env.OTEL_SERVICE_NAME:llama-stack}
+          sinks: ${env.TELEMETRY_SINKS:console, otel_trace, sqlite} <-add otel_trace and/or otel_metric
+          otel_trace_endpoint: ${env.OTEL_TRACE_ENDPOINT:} <-add ONLY if opentelemetry receiver endpoint is available.
+---
+```
+And, in [kubernetes/llama-stack/deployment.yaml](../llama-stack/deployment.yaml)
+
+```yaml
+---
+        env:
+        - name: OTEL_SERVICE_NAME
+          value: llamastack
+        - name: OTEL_TRACE_ENDPOINT
+          value: http://otel-collector-collector.observability-hub.svc.cluster.local:4318/v1/traces
+       #-  name: OTEL_METRIC_ENDPOINT
+       #-  value: http://otel-collector-collector.observability-hub.svc.cluster.local:4318/v1/metrics
+---
+```
+
+The otel-endpoint is `http://service-name-otc.namespace-of-otc.svc.cluster.local:4318/v1/traces,metrics` if exporting to
+central otel-collector. If using otel-collector sidecar, this would be `http://localhost:4318/v1/traces`.
+
 ## OpenShift Observability Operators
 
 Operators are available from OperatorHub