Celery.Hostname getting overridden by Celery.Origin #3096
Open
Description
Describe your environment
OS: (e.g, Ubuntu)
Python version: (e.g., Python 3.8.10)
Package version: (e.g., 0.46.0)
What happened?
When using Celery in python for task scheduling and instrumenting with
I think there's a bug, we are adding correct hostname in metrics but are adding origin in spans attributes.
However, origin is the celery worker thread id eg gen8@e338b12745fd This ID is defined by concurrency level. Eg if concurrency=11, there would be 11 such origins.
eg:
1 celery worker -> 1 host id: celery@66c3e95a829e
concurrency=11 -> 11 origins: gen8@e338b12745fd , gen8@38813890dc6e and so on.
in spans both this should be appended i.e. origin and worker_id
relavent code:
Celery Host:
-------------- celery@celery@0cc215850a9f v5.4.0 (opalescent)
--- ***** -----
-- ******* ---- Linux-6.10.14-linuxkit-aarch64-with 2024-12-12 13:27:37
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: myproject:0xffff91429160
- ** ---------- .> transport: amqp://guest:**@rabbitmq:5672//
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 11 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> queue1 exchange=queue1(direct) key=queue1
.> queue2 exchange=queue2(direct) key=queue2
.> queue3 exchange=queue3(direct) key=queue3
.> queue4 exchange=queue4(direct) key=queue4
Stack Trace Dump from Celery:
[2024-12-12 13:05:20,545: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0xffffb9aacd30> (args:('tasks.tasks.add', '0be9c7b8-1660-4a02-a644-93760ba96432', {'argsrepr': '[1, 2]', 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'id': '0be9c7b8-1660-4a02-a644-93760ba96432', 'ignore_result': False, 'kwargsrepr': '{}', 'lang': 'py', 'origin': 'gen8@8045f5fa26ea', 'parent_id': None, 'replaced_task_nesting': 0, 'retries': 0, 'root_id': '0be9c7b8-1660-4a02-a644-93760ba96432', 'shadow': None, 'stamped_headers': None, 'stamps': {}, 'task': 'tasks.tasks.add', 'timelimit': [None, None], 'traceparent': '00-c543e77303aa032d7de8c587383e1b26-9c8f7a55695e896b-01', 'properties': {'content_type': 'application/json', 'content_encoding': 'utf-8', 'application_headers': {'argsrepr': '[1, 2]', 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'id': '0be9c7b8-1660-4a02-a644-93760ba96432', 'ignore_result': False, 'kwargsrepr': '{}', 'lang': 'py', 'origin': 'gen8@8045f5fa26ea', 'parent_id': None, 'replaced_task_nesting': 0, 'retries': 0, 'root_id': '0be9c7b8-1660-4a02-a644-93760ba96432',... kwargs:{})
Current span attribute
Resource SchemaURL:
2024-12-12T13:07:42.890504760Z Resource attributes:
2024-12-12T13:07:42.890506343Z -> telemetry.sdk.language: Str(python)
2024-12-12T13:07:42.890507927Z -> telemetry.sdk.name: Str(opentelemetry)
2024-12-12T13:07:42.890522552Z -> telemetry.sdk.version: Str(1.29.0)
2024-12-12T13:07:42.890529427Z -> service.name: Str(worker1)
2024-12-12T13:07:42.890537302Z -> telemetry.auto.version: Str(0.50b0)
2024-12-12T13:07:42.890539010Z ScopeSpans #0
2024-12-12T13:07:42.890540510Z ScopeSpans SchemaURL:
2024-12-12T13:07:42.890545385Z InstrumentationScope opentelemetry.instrumentation.celery 0.50b0
2024-12-12T13:07:42.890549677Z Span #0
2024-12-12T13:07:42.890551135Z Trace ID : a8f0f9545959493125be4bc61fc8919b
2024-12-12T13:07:42.890552677Z Parent ID : fe3ada56039d9189
2024-12-12T13:07:42.890554427Z ID : 307d552d67686a83
2024-12-12T13:07:42.890555927Z Name : run/tasks.tasks.add
2024-12-12T13:07:42.890563010Z Kind : Consumer
2024-12-12T13:07:42.890565010Z Start time : 2024-12-12 13:07:40.671230259 +0000 UTC
2024-12-12T13:07:42.890566677Z End time : 2024-12-12 13:07:42.674325802 +0000 UTC
2024-12-12T13:07:42.890568260Z Status code : Unset
2024-12-12T13:07:42.890569760Z Status message :
2024-12-12T13:07:42.890571552Z Attributes:
2024-12-12T13:07:42.890573052Z -> celery.action: Str(run)
2024-12-12T13:07:42.890574718Z -> celery.state: Str(SUCCESS)
2024-12-12T13:07:42.890576302Z -> messaging.conversation_id: Str(f2c34be5-c865-471c-838c-e3e4810586c2)
2024-12-12T13:07:42.890577927Z -> messaging.destination: Str(queue1)
2024-12-12T13:07:42.890579552Z -> celery.delivery_info: Str({'exchange': '', 'routing_key': 'queue1', 'priority': 0, 'redelivered': False})
2024-12-12T13:07:42.890581343Z -> messaging.message.id: Str(f2c34be5-c865-471c-838c-e3e4810586c2)
2024-12-12T13:07:42.890583135Z -> celery.reply_to: Str(1c98ab71-800e-367c-88cb-78d0c648bc17)
2024-12-12T13:07:42.890584885Z -> celery.hostname: Str(gen8@8045f5fa26ea)
2024-12-12T13:07:42.890586468Z -> celery.task_name: Str(tasks.tasks.add)
2024-12-12T13:07:42.890588010Z {"kind": "exporter", "data_type": "traces", "name": "debug"}
Metric span attribute for hostname
2024-12-12T13:10:48.765017096Z Timestamp: 2024-12-12 13:10:48.761 +0000 UTC
2024-12-12T13:10:48.765018888Z Value: 1734008190.340000
2024-12-12T13:10:48.765025013Z Metric #1
2024-12-12T13:10:48.765027929Z Descriptor:
2024-12-12T13:10:48.765029263Z -> Name: flower_events_total
2024-12-12T13:10:48.765030554Z -> Description: Number of events
2024-12-12T13:10:48.765031846Z -> Unit:
2024-12-12T13:10:48.765033096Z -> DataType: Sum
2024-12-12T13:10:48.765034346Z -> IsMonotonic: true
2024-12-12T13:10:48.765035596Z -> AggregationTemporality: Cumulative
2024-12-12T13:10:48.765040679Z NumberDataPoints #0
2024-12-12T13:10:48.765047263Z Data point attributes:
2024-12-12T13:10:48.765087429Z -> type: Str(task-succeeded)
2024-12-12T13:10:48.765138013Z -> worker: Str(celery@0cc215850a9f)
2024-12-12T13:10:48.765153013Z StartTimestamp: 2024-12-12 12:59:38.75 +0000 U
Steps to Reproduce
Run Celery and use OTeL instrumentation (either SDK or Auto instrumentation)
Toy example to try out: https://github.com/shivanshuraj1333/celery-opentelemetry-instrumentation
Run:
docker-compose up
and checkout metrics and span attributes
Expected Result
celery.hostname should be consistent in spans and traces, and should not be confused with origin
Actual Result
celery.hostname should show actual hostname that is being shown in metrics as well
Additional context
No response
Would you like to implement a fix?
None