OpenTelemetry collector mode#1535
Conversation
|
✔️ All good! |
There was a problem hiding this comment.
I've attempted to clarify and make explicit some decisions contained within this in the comments. Other decisions to point out, many of which were also present in #1515:
- The agent still runs. This provides host metrics, and preserves compatibility with out StatsD and NGINX metrics support. I believe this is a good idea and would like to backport it to Python's collector mode.
- The OpenTelemetry gems must be installed manually in order to keep compatibility with Ruby 2.7, for which no compatible version of the OpenTelemetry gems exists. This is a significant hurdle for the installation process. We can soften it with a meta-gem that installs those dependencies, so that at least it's still a one-liner. We can also bump the gem's Ruby version requirement to Ruby 3.1, which regardless of the gem's installation requirement, is a requirement for collector mode as currently implemented.
- The oddities and differences in how OpenTelemetry trace context propagation is done in the background job libraries mirror the oddities of the corresponding OpenTelemetry contrib instrumentations, or their closest equivalent. This attempts to provide wire compatibility between the two, such that you could enqueue a background job in a collector mode AppSignal-instrumented application, process it in an OpenTelemetry-instrumented application pointed at the same collector, and have the traces link correctly.
| otel_attributes["appsignal.group"] = group.to_s | ||
| otel_attributes["appsignal.format"] = FORMAT_NAMES.fetch(format, "autodetect") |
There was a problem hiding this comment.
Support for these attributes was implemented in https://github.com/appsignal/appsignal-collector/pull/384 and https://github.com/appsignal/appsignal-processor-rs/pull/2138.
| # before encountering an error. | ||
| # | ||
| # Only the last 20 added breadcrumbs will be saved. | ||
| # At most 20 of the added breadcrumbs will be saved. |
There was a problem hiding this comment.
This is a genuine divergence in behaviour between collector mode and agent mode: agent mode saves the last 20, while collector mode saves the first 20. We could choose to make them both converge on the collector mode's behaviour, but it would be a breaking change for the agent.
| # grouping key available for later filtering. | ||
| def write_event_name_attributes(span, name, title) | ||
| span.name = title && !title.empty? ? title : name | ||
| span.set_attribute("appsignal.category", name) |
There was a problem hiding this comment.
This does not have any specific purpose in our OpenTelemetry UI, but will be shown as an internal attribute.
| if SPAN_KIND_BY_NAMESPACE.fetch(@namespace, DEFAULT_SPAN_KIND) == :consumer | ||
| "appsignal.function.parameters" | ||
| else | ||
| "appsignal.request.payload" | ||
| end |
There was a problem hiding this comment.
This is a bit of a hack -- it makes the not-necessarily-true assumption that the "params" in transactions with consumer root spans (from background jobs being processed) will represent function parameters, and otherwise they will represent a request payload.
Since agent mode didn't really have this differentiation to begin with, I think it's fine. We could choose to expose explicit setters for each, but I think that's beyond the scope of this work at the moment.
| if remote && kind == :server | ||
| tracer.start_span(name, :with_parent => opentelemetry_context, :kind => kind) | ||
| elsif remote && kind == :consumer | ||
| tracer.start_root_span( | ||
| name, | ||
| :kind => kind, | ||
| :links => [::OpenTelemetry::Trace::Link.new(remote)] | ||
| ) | ||
| else | ||
| tracer.start_root_span(name, :kind => kind) | ||
| end |
There was a problem hiding this comment.
Another sneaky assumption: when an OpenTelemetry trace context is passed when creating a transaction, when the transaction has a "server-like" namespace, then the root span of that transaction will be the child of that trace context's span, and when the transaction has a "consumer-like" namespace, then the root span of the transaction will have a span link to that trace context's spans. This matches the difference in behaviour between background job libraries and HTTP client/server libraries in the OpenTelemetry Ruby ecosystem.
At the Transaction API level, for custom instrumentation, we probably would want to expose an explicit flag to choose the kind of relationship. But generally the assumption with collector mode is that this is a compatibility layer, and newly implemented custom instrumentation would use the OpenTelemetry SDK directly, so this wasn't a priority.
There was a problem hiding this comment.
Note that, when creating a transaction's span, it's either a root span, or it's a parent of the span from the remote trace context -- it's never a child of the current span in the active OpenTelemetry context. This ensures that transactions always translate to traces, and aren't accidentally nested as spans inside the traces created by an external OpenTelemetry instrumentation.
| @span.add_event( | ||
| "appsignal.queue_start", | ||
| :timestamp => Time.at(start / 1000.0), | ||
| :attributes => { "appsignal.queue_start" => start } | ||
| ) |
There was a problem hiding this comment.
In addition to emitting the metrics that the agent would derive from the queue start, we also emit the queue start time itself as an event on the root span.
| # The open event span, or the root span when no event is open. Not the OTel | ||
| # current span, which may belong to another instrumentation. | ||
| def current_span | ||
| span, _token = @event_stack.last | ||
| span || @span | ||
| end |
There was a problem hiding this comment.
The "golden rule" of the AppSignal-to-OpenTelemetry interoperability story is that we don't write AppSignal-flavoured attributes and events to whatever is the current span in the active OpenTelemetry context, preventing AppSignal from "polluting" spans from external OpenTelemetry instrumentations. Instead, we keep a reference to the last AppSignal span created by the transaction: either the transaction's root span, or the span for the currently active event.
This way, whenever we add AppSignal-flavoured events or attributes, we always add them to one of our spans, either by using this helper or by referencing @span (the transaction's root span) directly -- we never add them to the current span in the active OpenTelemetry context.
| def start_event(opentelemetry_kind: nil) | ||
| span = tracer.start_span(EVENT_SPAN_PLACEHOLDER_NAME, :kind => opentelemetry_kind) |
There was a problem hiding this comment.
Note that we don't pass an explicit current_span parent when creating the span for an event -- we allow the span an AppSignal event to be a child of whichever the current active span in the OpenTelemetry context is. This allows AppSignal events and OpenTelemetry spans to nest within each other.
| def otel_header_name(env_key) | ||
| if env_key.start_with?("HTTP_") | ||
| env_key.delete_prefix("HTTP_").downcase.tr("_", "-") | ||
| elsif env_key.start_with?("CONTENT_") | ||
| env_key.downcase.tr("_", "-") | ||
| end | ||
| end |
There was a problem hiding this comment.
This is a divergence between agent mode and collector mode, in that the "Environment" section in agent mode contains other Rack CGI-flavoured attributes in it as well as HTTP headers, but in collector mode, only actual HTTP headers will be emitted as http.request.header.* attributes, and all other CGI-flavoured attributes in it from Rack will be dropped.
| # stubs Excon). Verifies, end to end, that the inject-only middleware writes | ||
| # trace context onto a live outgoing request while the instrumentor's client | ||
| # event span is current -- the ordering the stubbed suite can't prove. | ||
| describe "Excon integration" do |
There was a problem hiding this comment.
The existing Excon integration was previously untested -- this tests it just enough to assert that trace context propagation takes place.
| def report_errors | ||
| if @backend.records_errors_eagerly? | ||
| run_error_blocks | ||
| else | ||
| report_errors_as_duplicates | ||
| end | ||
| end | ||
|
|
||
| # Eager mode: the errors are already recorded, so just run their blocks. | ||
| # Blocks run in add-order, so a later error's block wins on a shared key, and | ||
| # all block-set metadata merges onto the root span. (Per-error metadata | ||
| # isolation is deferred -- the processor/UI does not read per-event | ||
| # attributes yet.) | ||
| def run_error_blocks | ||
| @error_blocks.each_value do |blocks| | ||
| self.class.with_transaction(self) do | ||
| blocks.each { |block| block.call(self) } | ||
| end | ||
| end | ||
| end |
There was a problem hiding this comment.
I'm realising now that this is likely not the right behaviour. We probably want to run error blocks immediately in collector mode. Unlike in agent mode, where we need to store them in order to fabricate duplicates at completion time, I see no benefit to storing them and running them later in collector mode.
I don't think error blocks should be used like this (in collector mode, they likely should not be used at all) but if the block contains things that should be attached to the current span, such as breadcrumbs (🌵), other errors (😨) or custom OpenTelemetry instrumentation (💀) then, by running them immediately, those would be attached to the right point in the span tree.
(to be clear: none of those are things that an error block should contain)
There was a problem hiding this comment.
Fixed in c8fbb04.
The one non-obvious part is @error_blocks. Collector mode no longer stores the block there, but it still registers the error as a key, because we rely on those keys for last_errors, the distinct-error limit, and the duplicate check. And since blocks now run when added, report_errors only has work to do in agent mode, so run_error_blocks is gone.
Added a collector-mode test to lock it in: an error whose block adds a breadcrumb, asserting the breadcrumb lands on the event span, not the root. Fails without the change.
This comment has been minimized.
This comment has been minimized.
31d2771 to
c220cce
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
1 similar comment
This comment has been minimized.
This comment has been minimized.
Introduce an optional "collector mode": when `collector_endpoint` (or `APPSIGNAL_COLLECTOR_ENDPOINT`) is set, `Appsignal.start` boots an OpenTelemetry SDK that exports OTLP/HTTP to that endpoint alongside the usual C-extension agent. No AppSignal-collected data routes through the SDK yet — this commit stands up the mode, its configuration, and the seams the later commits build on. The OpenTelemetry gems stay optional (not gemspec dependencies): the collector boot-checks for them and falls back to the agent when they are absent or too old. Configuration gains the collector attribute options (service name, filter/send/response settings) that shape the exported resource. Also adds the shared test infrastructure the rest of the work relies on: the `-collector` gemfile variants, an in-memory OTLP mock server, and the agent/collector dual-mode example helpers.
Route `Appsignal::Logger` through a backend seam: the extension backend keeps the existing C-extension logging, and a new OpenTelemetry backend emits log records through the SDK's logger provider when collector mode is active. `Appsignal::Backends.logger` picks between them. The OpenTelemetry backend maps Ruby severities and formats to their OTLP equivalents and attaches `appsignal.group`/`appsignal.format` as hard overrides so user attributes can't spoof them.
Route the custom metric helpers (`set_gauge`, `increment_counter`, `add_distribution_value`) through `Appsignal::Backends.metrics`: the extension backend keeps the C-extension path (including the out-of-range warning), and a new OpenTelemetry backend records gauges, up/down counters and histograms through the SDK's meter provider when collector mode is active. With both the metrics and logger backends in place, `Appsignal.start` can fall back to the agent when the OpenTelemetry SDK fails to boot, so telemetry is never silently dropped.
Dual-mode the GVL, MRI and Sidekiq probe specs so each metric example runs in both agent and collector mode. The probes emit through the metric helpers, so no probe code changes — this just proves their output routes to OpenTelemetry when collector mode is active.
Back `Appsignal::Transaction` with a pluggable backend. The extension backend keeps the existing C-extension behaviour (it now owns the sample-data serialization, breadcrumb buffering and the agent's error-cause shape). A new OpenTelemetry backend maps a transaction to an OTel root span and each event to a child span, chosen by `Appsignal::Backends.transaction`. Because one `Transaction` drives two error models, the backend exposes `records_errors_eagerly?`: the extension collects errors then duplicates the transaction per error at completion, while the OTel backend records each as an `exception` span event straight away. Key mapping decisions, for reviewers: - Span kind comes from the namespace (web/action_cable => SERVER, background_job => CONSUMER); datastore client events pass `opentelemetry_kind: :client`, set at creation since the kind is immutable. - An event span's name is `title || name`, with the event name kept as `appsignal.category`; SQL bodies map to `db.query.text`, everything else to `appsignal.body`. - Errors attach to the currently-open span with the cause chain in one `appsignal.error_causes` JSON attribute; a discarded transaction sets `appsignal.ignore_subtrace` so the collector drops it.
Dual-mode the hook, integration, Rack and loader specs so each existing event-emission example runs in both agent and collector mode, asserting the OpenTelemetry span shape (name, category, body, kind, parent) in collector mode. No library changes — this exercises the trace backend through every integration. Trace-context propagation (incoming extraction, outgoing injection) is left to the propagation commits, so the specs that also cover it are dual-moded there alongside their library changes.
In collector mode, HTTP client integrations write the current W3C trace context onto their outgoing requests so the downstream service continues the same trace. Net::HTTP, HTTP.rb, Excon and Faraday each inject a `traceparent` header (Excon via a dedicated middleware, HTTP.rb on every hop), and mark the request span with CLIENT kind. Injection is a no-op outside collector mode, so agent-mode behaviour is unchanged. Excon moves to its own gemfile now that it is exercised directly rather than only through Faraday's adapter.
In collector mode, the Rack middleware, Rack event handler and the Webmachine integration read an incoming W3C `traceparent` header and pass it to the transaction. A web (SERVER) transaction continues under the remote span; a job (CONSUMER) transaction starts its own trace linked back to it; with no context, or outside collector mode, it stays a plain root span. `Transaction.create` gains an `opentelemetry_context:` keyword that threads through to the backend, which parents or links the root span accordingly.
Carry the W3C trace context across the enqueue/perform boundary for Sidekiq, Resque, Que, Shoryuken and Active Job. On enqueue (in collector mode) each client writes the current context onto the job payload and marks the enqueue span with PRODUCER kind; on perform the server side reads it back and links the job's trace to the enqueuing span. Each library carries the context on its own channel: a job hash key, an SQS message attribute, a Que tag, or Active Job's serialized arguments. It skips propagation rather than breaking the enqueue when a channel is full, and is a no-op outside collector mode.
In collector mode each error is recorded when it is added, but its block was still held until completion. Deferring only makes sense in agent mode, where blocks run against the duplicate transactions fabricated at completion. In collector mode there is no benefit to holding them back. Run the block when the error is added instead. Side effects that target the current span, such as breadcrumbs, nested errors or custom instrumentation, then land where the error was reported rather than on the root span at completion. None of those belong in an error block, but this puts them in the right place if they are used. With blocks no longer deferred in collector mode, the error tracking that supported deferral was doing two jobs at once. `@error_blocks` was both the set of distinct errors, which drives `last_errors`, the dedup check and the error limit, and the per-error blocks that only agent mode runs. Split them. `@errors` is now the distinct-error set, used in both modes, and `@error_blocks` holds only blocks and is populated only in agent mode. `@errors` is a Set, so membership is by object identity, matching how the errors were deduplicated as hash keys before. Finally, rename `records_errors_eagerly?` to `supports_multiple_errors?`. The old name described the mechanism. The real question at the call site is whether the backend can hold more than one error on a transaction.
The sequel-rails gem emits its queries as `sql.sequel` ActiveSupport::Notifications events. Those are recorded through the generic notifications integration, not the dedicated Sequel hook, so the hook's CLIENT-kind tagging never applied to them. The notifications integration only listed `sql.active_record` as an outgoing datastore call, so Sequel queries were exported with the default (INTERNAL) span kind in collector mode. Add `sql.sequel` to that list so a Sequel query is CLIENT regardless of which path records it, matching ActiveRecord and the dedicated Sequel hook.
The `extract_rack_context` and `extract_job_context` helpers called `OpenTelemetry.propagation.extract` without a `context:` argument, so it defaulted to `Context.current`. The W3C extractor returns its base context unchanged when the carrier has no `traceparent`, so a request with no incoming trace context inherited whatever span happened to be attached to the fiber. In collector mode that let a span left over from an earlier request become the parent of a later request's root span, merging unrelated requests into a single trace. Extract onto `Context.empty` instead, so extraction reflects only the carrier. A real incoming `traceparent` still parents (web) or links (jobs) exactly as before.
A transaction runs blocks supplied by user code in a few places: the error block passed to `set_error`, `send_error`, and `report_error`, and the `after_create` and `before_complete` hooks. Any of them can raise. These blocks can run far from where they were defined. In agent mode an error block runs at completion, not when the error is added. So a raise propagated out of whatever drove creation or completion, surfacing the error in unrelated code. In collector mode it also skipped the backend's completion step, which left the transaction's OpenTelemetry context attached to the fiber. That leaked context then became the parent of the next request's root span on the reused thread, merging unrelated requests into one trace. Agent mode never showed the leak because its extension handle is not a per-fiber stack. Wrap each error block once, where user code hands it in, so it stays protected wherever it later runs. `send_error` and `report_error` name themselves when they wrap the block, so the log points the customer at the call they made rather than at an internal method name. The `after_create` and `before_complete` hooks are guarded where they run, because they can be registered with a block or by pushing a method onto the set, so there is no single entry point to wrap them at. On failure, log the error and the block's definition site, then carry on so creation and completion always finish. The error is swallowed rather than re-raised, because the block can run far from its caller and re-raising would surface it in code that has nothing to do with it.
A web request that never sets an action name (for example a static asset served without a controller) has no action to group by. Agent mode does not report such a transaction. In collector mode the root span is created with a placeholder name, `appsignal.transaction <namespace>`, so every actionless request was surfacing under that one shared placeholder action. Track whether an action was set and, when it was not, flag the root span with `appsignal.ignore_subtrace` on completion, exactly as `discard` does. The AppSignal Collector (0.10.0 and newer) then drops the subtrace, so these requests no longer appear, matching agent mode.
Enqueuing a Delayed Job now records an `enqueue.delayed_job` event on the active transaction, so enqueues made from a web request or another job show up in the event timeline. Delayed Job was the only background job backend still missing this. In collector mode that event also opens a producer span, matching the other job backends and OpenTelemetry's own Delayed Job instrumentation. Delayed Job has no envelope to carry trace context across the enqueue/perform boundary, so -- like OpenTelemetry -- nothing is injected and the producer and consumer spans are not linked. Test it against Delayed Job's in-memory test backend under a new `delayed_job` gemfile, driving the real enqueue and perform lifecycle in both agent and collector mode.
Audit of the transaction and helper specs for behaviors covered in only one mode, agent or collector, that should be covered in both. Most gaps were agent-only leftovers that assert through `to_h`-based matchers, which do nothing in collector mode. Fold the two send_error block specs into one dual-moded spec. It sets an active transaction, calls `send_error` with a block, and checks the block's metadata and the error land on `send_error`'s own transaction while the active one is restored and left untouched. In collector mode this confirms the active trace continues rather than being dropped or left open. Cover more behaviors in both modes: the `set_error` and `report_error` blocks, `set_empty_params!`, the `add_params`, `add_session_data`, `add_headers` and `add_custom_data` merge and block variants, `set_action` and `set_namespace` setting the value, `report_error` not completing the active transaction, and a transaction that reports a cause-less error after one with causes. Also add the one missing collector metric-coercion case, a symbol metric name, to the OpenTelemetry metrics backend spec.
This is a message from the daily scheduled checks. |
Somewhat reviewable re-do of #1515.
While it's still a massive amount of changes, it separates the signal (implementing collector mode for metrics, traces and logs, as well as adding trace context propagation to our existing library and framework integrations) from the noise (making each and every integration's existing tests run both in agent mode and collector mode)
Add OpenTelemetry collector mode
Introduce an optional "collector mode": when
collector_endpoint(orAPPSIGNAL_COLLECTOR_ENDPOINT) is set,Appsignal.startboots anOpenTelemetry SDK that exports OTLP/HTTP to that endpoint alongside the
usual C-extension agent. No AppSignal-collected data routes through the
SDK yet — this commit stands up the mode, its configuration, and the
seams the later commits build on.
The OpenTelemetry gems stay optional (not gemspec dependencies):
collector mode boot-checks for them and falls back to the agent when they
are absent or too old. Configuration gains the collector attribute
options (service name, filter/send/response settings) that shape the
exported resource.
Also adds the shared test infrastructure the rest of the work relies on:
the
-collectorgemfile variants, an in-memory OTLP mock server, and theagent/collector dual-mode example helpers.
Emit OpenTelemetry logs in collector mode
Route
Appsignal::Loggerthrough a backend seam: the extension backendkeeps the existing C-extension logging, and a new OpenTelemetry backend
emits log records through the SDK's logger provider when collector mode
is active.
Appsignal::Backends.loggerpicks between them.The OpenTelemetry backend maps Ruby severities and formats to their OTLP
equivalents and attaches
appsignal.group/appsignal.formatas hardoverrides so user attributes can't spoof them.
Emit OpenTelemetry metrics in collector mode
Route the custom metric helpers (
set_gauge,increment_counter,add_distribution_value) throughAppsignal::Backends.metrics: theextension backend keeps the C-extension path (including the out-of-range
warning), and a new OpenTelemetry backend records gauges, up/down
counters and histograms through the SDK's meter provider when collector
mode is active.
With both the metrics and logger backends in place,
Appsignal.startcan fall back to the agent when the OpenTelemetry SDK fails to boot, so
telemetry is never silently dropped.
Test the metric probes in collector mode
Dual-mode the GVL, MRI and Sidekiq probe specs so each metric example
runs in both agent and collector mode. The probes emit through the
metric helpers, so no probe code changes — this just proves their
output routes to OpenTelemetry when collector mode is active.
Emit OpenTelemetry traces in collector mode
Back
Appsignal::Transactionwith a pluggable backend. The extensionbackend keeps the existing C-extension behaviour (it now owns sample-data
serialization, breadcrumb buffering and the agent's error-cause shape).
A new OpenTelemetry backend maps a transaction to an OTel root span and
each event to a child span, chosen by
Appsignal::Backends.transaction.Because one
Transactiondrives two error models, the backend exposesrecords_errors_eagerly?: the extension collects errors then duplicatesthe transaction per error at completion, while the OTel backend records
each as an
exceptionspan event straight away.Key mapping decisions, for reviewers:
background_job => CONSUMER); datastore client events pass
opentelemetry_kind: :client, set at creation since kind is immutable.title || name, with the event name kept asappsignal.category; SQL bodies map todb.query.text, everythingelse to
appsignal.body.appsignal.error_causesJSON attribute; a discarded transaction setsappsignal.ignore_subtraceso the collector drops it.Test integrations in collector mode
Dual-mode the hook, integration, Rack and loader specs so each existing
event-emission example runs in both agent and collector mode, asserting
the OpenTelemetry span shape (name, category, body, kind, parent) in
collector mode. No library changes — this exercises the trace backend
through every integration.
Trace-context propagation (incoming extraction, outgoing injection) is
left to the propagation commits, so the specs that also cover it are
dual-moded there alongside their library changes.
Inject trace context into outgoing HTTP requests
In collector mode, HTTP client integrations write the current W3C trace
context onto their outgoing requests so the downstream service continues
the same trace. Net::HTTP, HTTP.rb, Excon and Faraday each inject a
traceparentheader (Excon via a dedicated middleware, HTTP.rb on everyhop), and mark the request span with CLIENT kind.
Injection is a no-op outside collector mode, so agent-mode behaviour is
unchanged. Excon moves to its own gemfile now that it is exercised
directly rather than only through Faraday's adapter.
Extract trace context from incoming web requests
In collector mode, the Rack middleware, Rack event handler and Webmachine
integration read an incoming W3C
traceparentheader and pass it to thetransaction. A web (SERVER) transaction continues under the remote span;
a job (CONSUMER) transaction starts its own trace linked back to it; with
no context, or outside collector mode, it stays a plain root span.
Transaction.creategains anopentelemetry_context:keyword thatthreads through to the backend, which parents or links the root span
accordingly.
Propagate trace context across background jobs
Carry the W3C trace context across the enqueue/perform boundary for
Sidekiq, Resque, Que, Shoryuken and Active Job. On enqueue (in collector
mode) each client writes the current context onto the job payload and
marks the enqueue span with PRODUCER kind; on perform the server side
reads it back and links the job's trace to the enqueuing span.
Each library carries the context on its own channel — a job hash key, an
SQS message attribute, a Que tag, or Active Job's serialized arguments —
skipping propagation rather than breaking the enqueue when a channel is
full. It's a no-op outside collector mode.