feat: 给 asynctaskext/busext 增加 OpenTelemetry 链路追踪埋点 - #745
Open
guohuachan wants to merge 3 commits into
Open
Conversation
此前 OTEL_ENABLE 只让 gobay 初始化 TracerProvider/exporter,异步侧 (machinery worker、bus 消费/发布)没有任何创建 span 的代码,asynctask 和 bus 在 otel 里始终零数据。本次补齐消息链路两端的插桩: asynctaskext: - SendTaskWithContext 创建 SpanKind=Producer 的 "send/<task>" span, 并把 traceparent 注入 Signature.Headers(machinery 随消息序列化) - worker 的 Pre/PostTaskHandler 按 signature.UUID 记录 SpanKind=Consumer 的 "run/<task>" span:消息头带 traceparent 时接续 上游 trace,否则自建 root;与耗时指标同理,machinery 全局 hook 拿不到 该次调用的 error,span 状态保持 Unset busext: - 新增 PushWithContext:Producer span + 注入 traceparent 到消息 Headers,原 Push 行为不变 - dispatch 包一层 Consumer span("run/<routingKey>"),按结果状态收 尾:非 success 标记 Error,失败的 bus 事件因此能被 collector 的 tail_sampling 保留 observability 新增 MapCarrier,把 tasks.Headers / amqp.Table 适配成 TextMapCarrier。全部埋点以 OTEL_ENABLE 环境变量为开关(与现有 initOtel 同源),未开启时零行为变化。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6xmHTyA1Wfgcf3igefjU3
吸收 shanbay#741 中 shanbay#745 缺少的两块能力: - HandlerWithContext 可选接口:实现 RunWithContext(ctx) 的 handler 在消费 时拿到携带上游 trace 上下文(及本次 Consumer span)的 ctx,传给 ent/redis/RPC 即可整链贯通;未实现的 handler 走 Run() 零改动。接口分发 不依赖 OTEL_ENABLE,未开启时 ctx 为 Background - Consumer/Producer span 补 messaging.system / messaging.destination.name / messaging.operation 语义属性,Consumer 额外记 gobay.bus.status busext 的消费/发布插桩本体(dispatch Consumer span、PushWithContext) 两个 PR 重复,以本分支实现为准。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6xmHTyA1Wfgcf3igefjU3
3 tasks
该测试把 metrics server 放在 goroutine 里 ListenAndServe,之后立刻 http.Get,没有任何就绪等待;Get 抢在监听建立前发出时返回 connection refused,而 t.Error 不中止执行,随后 resp.Body.Close() 对 nil 解引用 直接 SIGSEGV(CI run 32938502153 的 Golang 1.24 job 实际命中)。 修复:启动 goroutine 后轮询端口可连接再继续;Get 出错改 t.Fatal, 杜绝 nil resp 解引用。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6xmHTyA1Wfgcf3igefjU3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
OTEL_ENABLE开启后 gobay 只初始化了 TracerProvider / OTLP exporter,span 只由 HTTP/gRPC/Redis 等请求路径上的中间件产生。asynctask(machinery worker)和 bus(AMQP 消费/发布)两条消息路径上没有任何创建 span 的代码——线上验证(以 quest 为例):quest.xyz一周 50w+ 条 api/rpc span,Consumer/Producer span 为 0。改动
asynctaskext
SendTaskWithContext:创建SpanKind=Producer的send/<task>span,并把traceparent注入Signature.Headers(machinery 会随消息序列化到 broker)Pre/PostTaskHandler:按signature.UUID记录SpanKind=Consumer的run/<task>span——消息头带traceparent时接续上游 trace,否则自建 root trace。与耗时指标(feat: 给 asynctaskext/busext 增加 Prometheus 处理耗时+QPS 埋点 #736)同理,machinery 全局 hook 拿不到该次调用的 error,span 状态保持 Unsetbusext
PushWithContext:Producer span + 注入traceparent到消息 Headers;原Push签名与行为不变。Python 端 celery worker(opentelemetry-instrumentation-celery)可自动续链dispatch包一层 Consumer span(run/<routingKey>),按结果状态收尾:非success标记Error——失败的 bus 事件因此能被 collector 的 tail_sampling 保留HandlerWithContext可选接口(自 feat: busext 接入 OTel trace(跨语言链路续接,基于 #740) #741):实现RunWithContext(ctx)的 handler 拿到携带上游 trace 上下文(及本次 Consumer span)的 ctx,传给 ent/redis/RPC 即整链贯通;未实现的 handler 走Run()零改动,接口分发不依赖OTEL_ENABLEmessaging.system/messaging.destination.name/messaging.operation语义属性,Consumer 额外记gobay.bus.status(自 feat: busext 接入 OTel trace(跨语言链路续接,基于 #740) #741)observability
MapCarrier:把tasks.Headers/amqp.Table(map[string]interface{})适配成 otelTextMapCarrier行为边界
OTEL_ENABLE环境变量为开关(与现有initOtel同源),未开启时零行为变化OTEL_ENABLE/OTEL_SERVICE_NAME)run/<name>),Python→Go / Go→Python 链路可互续;Python 消费侧 parent-based 采样的配套修复见 backend-lib/coast!242HandlerWithContext拿到 ctx测试
go test ./extensions/asynctaskext/ ./extensions/busext/(本地 redis + rabbitmq:3.8)全量通过,无回归🤖 Generated with Claude Code
https://claude.ai/code/session_01V6xmHTyA1Wfgcf3igefjU3