Description
What happened?
I'm setting up distributed tracing and end-to-end transaction monitoring for my services, which are hosted in an AKS (Azure Kubernetes Service) cluster and built using the Python framework. Since we use a microservices architecture, I'm implementing OpenTelemetry (OTel) with auto-instrumentation. This involves deploying the OTel Operator in AKS along with an OTel Collector and using Jaeger for tracing and ELK as backend storage
As part of the setup, I added the necessary Python-related annotations to enable tracing for four of my services. However, while I can see trace data for two of the services, the other two are not sending any data. Instead, I see an error message:
Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.
Since our environment is in Azure and not AWS, I don't understand why this error is appearing. What could be causing this issue, and how can I resolve it?
Steps to Reproduce
setup open telemetry by deploying otel operator in aks with this command
helm install opentelemetry-operator open-telemetry/opentelemetry-operator --namespace opentelemetry-operator --create-namespace --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"
--set admissionWebhooks.certManager.enabled=false
--set admissionWebhooks.autoGenerateCert.enabled=true
and deploy otel collector and instrumentation.yaml wity the following
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: traces
spec:
mode: deployment
config:
receivers:
otlp:
protocols:
grpc: {}
http: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
otlp:
endpoint: "jaeger-collector:4317"
endpoint: "jaeger-collector.observability.svc.cluster.local:4317"
endpoint: "jaeger-collector.jaeger.svc.cluster.local:4317"
tls:
insecure: true
debug: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, otlp]
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-instrumentation
spec:
exporter:
endpoint: http://traces-collector.opentelemetry-operator.svc.cluster.local:4317/
propagators:
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "1"
python:
env: - name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://traces-collector.opentelemetry-operator.svc.cluster.local:4318/v1/traces - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
value: aws-lambda,metrics - name: OTEL_METRICS_EXPORTER
value: none - name: OTEL_LOG_LEVEL
value: "debug"
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
Expected Result
need to get traces exported from the services of aks to the jaeger ui backend but not able to get that
Actual Result
i'm getting this error
Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.
this is aws labda error which is not related to aks or azure but i'm getting this error in the logs of services which i auto instrumenetd with these annotations
instrumentation.opentelemetry.io/inject-python:true
instrumentation.opentelemetry.io/otel-python-platform:musl
instrumentation.opentelemetry.io/container-names
Additional context
No response
Would you like to implement a fix?
None