Skip to content

OpenTelemetry collector mode#1535

Open
unflxw wants to merge 16 commits into
mainfrom
wip-otel-rework
Open

OpenTelemetry collector mode#1535
unflxw wants to merge 16 commits into
mainfrom
wip-otel-rework

Conversation

@unflxw

@unflxw unflxw commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Somewhat reviewable re-do of #1515.

While it's still a massive amount of changes, it separates the signal (implementing collector mode for metrics, traces and logs, as well as adding trace context propagation to our existing library and framework integrations) from the noise (making each and every integration's existing tests run both in agent mode and collector mode)

Add OpenTelemetry collector mode

Introduce an optional "collector mode": when collector_endpoint (or
APPSIGNAL_COLLECTOR_ENDPOINT) is set, Appsignal.start boots an
OpenTelemetry SDK that exports OTLP/HTTP to that endpoint alongside the
usual C-extension agent. No AppSignal-collected data routes through the
SDK yet — this commit stands up the mode, its configuration, and the
seams the later commits build on.

The OpenTelemetry gems stay optional (not gemspec dependencies):
collector mode boot-checks for them and falls back to the agent when they
are absent or too old. Configuration gains the collector attribute
options (service name, filter/send/response settings) that shape the
exported resource.

Also adds the shared test infrastructure the rest of the work relies on:
the -collector gemfile variants, an in-memory OTLP mock server, and the
agent/collector dual-mode example helpers.

Emit OpenTelemetry logs in collector mode

Route Appsignal::Logger through a backend seam: the extension backend
keeps the existing C-extension logging, and a new OpenTelemetry backend
emits log records through the SDK's logger provider when collector mode
is active. Appsignal::Backends.logger picks between them.

The OpenTelemetry backend maps Ruby severities and formats to their OTLP
equivalents and attaches appsignal.group/appsignal.format as hard
overrides so user attributes can't spoof them.

Emit OpenTelemetry metrics in collector mode

Route the custom metric helpers (set_gauge, increment_counter,
add_distribution_value) through Appsignal::Backends.metrics: the
extension backend keeps the C-extension path (including the out-of-range
warning), and a new OpenTelemetry backend records gauges, up/down
counters and histograms through the SDK's meter provider when collector
mode is active.

With both the metrics and logger backends in place, Appsignal.start
can fall back to the agent when the OpenTelemetry SDK fails to boot, so
telemetry is never silently dropped.

Test the metric probes in collector mode

Dual-mode the GVL, MRI and Sidekiq probe specs so each metric example
runs in both agent and collector mode. The probes emit through the
metric helpers, so no probe code changes — this just proves their
output routes to OpenTelemetry when collector mode is active.

Emit OpenTelemetry traces in collector mode

Back Appsignal::Transaction with a pluggable backend. The extension
backend keeps the existing C-extension behaviour (it now owns sample-data
serialization, breadcrumb buffering and the agent's error-cause shape).
A new OpenTelemetry backend maps a transaction to an OTel root span and
each event to a child span, chosen by Appsignal::Backends.transaction.

Because one Transaction drives two error models, the backend exposes
records_errors_eagerly?: the extension collects errors then duplicates
the transaction per error at completion, while the OTel backend records
each as an exception span event straight away.

Key mapping decisions, for reviewers:

  • Span kind comes from the namespace (web/action_cable => SERVER,
    background_job => CONSUMER); datastore client events pass
    opentelemetry_kind: :client, set at creation since kind is immutable.
  • An event span's name is title || name, with the event name kept as
    appsignal.category; SQL bodies map to db.query.text, everything
    else to appsignal.body.
  • Errors attach to the currently-open span with the cause chain in one
    appsignal.error_causes JSON attribute; a discarded transaction sets
    appsignal.ignore_subtrace so the collector drops it.

Test integrations in collector mode

Dual-mode the hook, integration, Rack and loader specs so each existing
event-emission example runs in both agent and collector mode, asserting
the OpenTelemetry span shape (name, category, body, kind, parent) in
collector mode. No library changes — this exercises the trace backend
through every integration.

Trace-context propagation (incoming extraction, outgoing injection) is
left to the propagation commits, so the specs that also cover it are
dual-moded there alongside their library changes.

Inject trace context into outgoing HTTP requests

In collector mode, HTTP client integrations write the current W3C trace
context onto their outgoing requests so the downstream service continues
the same trace. Net::HTTP, HTTP.rb, Excon and Faraday each inject a
traceparent header (Excon via a dedicated middleware, HTTP.rb on every
hop), and mark the request span with CLIENT kind.

Injection is a no-op outside collector mode, so agent-mode behaviour is
unchanged. Excon moves to its own gemfile now that it is exercised
directly rather than only through Faraday's adapter.

Extract trace context from incoming web requests

In collector mode, the Rack middleware, Rack event handler and Webmachine
integration read an incoming W3C traceparent header and pass it to the
transaction. A web (SERVER) transaction continues under the remote span;
a job (CONSUMER) transaction starts its own trace linked back to it; with
no context, or outside collector mode, it stays a plain root span.

Transaction.create gains an opentelemetry_context: keyword that
threads through to the backend, which parents or links the root span
accordingly.

Propagate trace context across background jobs

Carry the W3C trace context across the enqueue/perform boundary for
Sidekiq, Resque, Que, Shoryuken and Active Job. On enqueue (in collector
mode) each client writes the current context onto the job payload and
marks the enqueue span with PRODUCER kind; on perform the server side
reads it back and links the job's trace to the enqueuing span.

Each library carries the context on its own channel — a job hash key, an
SQS message attribute, a Que tag, or Active Job's serialized arguments —
skipping propagation rather than breaking the enqueue when a channel is
full. It's a no-op outside collector mode.

@backlog-helper

backlog-helper Bot commented Jul 8, 2026

Copy link
Copy Markdown

✔️ All good!

New issue guide | Backlog management | Rules | Feedback

@unflxw unflxw changed the title OpenTelemetry collector mode (narrative reconstruction of wip-opentelemetry) OpenTelemetry collector mode Jul 8, 2026
@unflxw
unflxw force-pushed the wip-otel-rework branch from ad053a7 to f62db5f Compare July 8, 2026 16:32

@unflxw unflxw left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've attempted to clarify and make explicit some decisions contained within this in the comments. Other decisions to point out, many of which were also present in #1515:

  • The agent still runs. This provides host metrics, and preserves compatibility with out StatsD and NGINX metrics support. I believe this is a good idea and would like to backport it to Python's collector mode.
  • The OpenTelemetry gems must be installed manually in order to keep compatibility with Ruby 2.7, for which no compatible version of the OpenTelemetry gems exists. This is a significant hurdle for the installation process. We can soften it with a meta-gem that installs those dependencies, so that at least it's still a one-liner. We can also bump the gem's Ruby version requirement to Ruby 3.1, which regardless of the gem's installation requirement, is a requirement for collector mode as currently implemented.
  • The oddities and differences in how OpenTelemetry trace context propagation is done in the background job libraries mirror the oddities of the corresponding OpenTelemetry contrib instrumentations, or their closest equivalent. This attempts to provide wire compatibility between the two, such that you could enqueue a background job in a collector mode AppSignal-instrumented application, process it in an OpenTelemetry-instrumented application pointed at the same collector, and have the traces link correctly.

Comment on lines +47 to +48
otel_attributes["appsignal.group"] = group.to_s
otel_attributes["appsignal.format"] = FORMAT_NAMES.fetch(format, "autodetect")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# before encountering an error.
#
# Only the last 20 added breadcrumbs will be saved.
# At most 20 of the added breadcrumbs will be saved.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a genuine divergence in behaviour between collector mode and agent mode: agent mode saves the last 20, while collector mode saves the first 20. We could choose to make them both converge on the collector mode's behaviour, but it would be a breaking change for the agent.

# grouping key available for later filtering.
def write_event_name_attributes(span, name, title)
span.name = title && !title.empty? ? title : name
span.set_attribute("appsignal.category", name)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not have any specific purpose in our OpenTelemetry UI, but will be shown as an internal attribute.

Comment on lines +390 to +394
if SPAN_KIND_BY_NAMESPACE.fetch(@namespace, DEFAULT_SPAN_KIND) == :consumer
"appsignal.function.parameters"
else
"appsignal.request.payload"
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a hack -- it makes the not-necessarily-true assumption that the "params" in transactions with consumer root spans (from background jobs being processed) will represent function parameters, and otherwise they will represent a request payload.

Since agent mode didn't really have this differentiation to begin with, I think it's fine. We could choose to expose explicit setters for each, but I think that's beyond the scope of this work at the moment.

Comment on lines +356 to +366
if remote && kind == :server
tracer.start_span(name, :with_parent => opentelemetry_context, :kind => kind)
elsif remote && kind == :consumer
tracer.start_root_span(
name,
:kind => kind,
:links => [::OpenTelemetry::Trace::Link.new(remote)]
)
else
tracer.start_root_span(name, :kind => kind)
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another sneaky assumption: when an OpenTelemetry trace context is passed when creating a transaction, when the transaction has a "server-like" namespace, then the root span of that transaction will be the child of that trace context's span, and when the transaction has a "consumer-like" namespace, then the root span of the transaction will have a span link to that trace context's spans. This matches the difference in behaviour between background job libraries and HTTP client/server libraries in the OpenTelemetry Ruby ecosystem.

At the Transaction API level, for custom instrumentation, we probably would want to expose an explicit flag to choose the kind of relationship. But generally the assumption with collector mode is that this is a compatibility layer, and newly implemented custom instrumentation would use the OpenTelemetry SDK directly, so this wasn't a priority.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that, when creating a transaction's span, it's either a root span, or it's a parent of the span from the remote trace context -- it's never a child of the current span in the active OpenTelemetry context. This ensures that transactions always translate to traces, and aren't accidentally nested as spans inside the traces created by an external OpenTelemetry instrumentation.

Comment on lines +139 to +143
@span.add_event(
"appsignal.queue_start",
:timestamp => Time.at(start / 1000.0),
:attributes => { "appsignal.queue_start" => start }
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to emitting the metrics that the agent would derive from the queue start, we also emit the queue start time itself as an event on the root span.

Comment on lines +330 to +335
# The open event span, or the root span when no event is open. Not the OTel
# current span, which may belong to another instrumentation.
def current_span
span, _token = @event_stack.last
span || @span
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "golden rule" of the AppSignal-to-OpenTelemetry interoperability story is that we don't write AppSignal-flavoured attributes and events to whatever is the current span in the active OpenTelemetry context, preventing AppSignal from "polluting" spans from external OpenTelemetry instrumentations. Instead, we keep a reference to the last AppSignal span created by the transaction: either the transaction's root span, or the span for the currently active event.

This way, whenever we add AppSignal-flavoured events or attributes, we always add them to one of our spans, either by using this helper or by referencing @span (the transaction's root span) directly -- we never add them to the current span in the active OpenTelemetry context.

Comment on lines +81 to +82
def start_event(opentelemetry_kind: nil)
span = tracer.start_span(EVENT_SPAN_PLACEHOLDER_NAME, :kind => opentelemetry_kind)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we don't pass an explicit current_span parent when creating the span for an event -- we allow the span an AppSignal event to be a child of whichever the current active span in the OpenTelemetry context is. This allows AppSignal events and OpenTelemetry spans to nest within each other.

Comment on lines +410 to +416
def otel_header_name(env_key)
if env_key.start_with?("HTTP_")
env_key.delete_prefix("HTTP_").downcase.tr("_", "-")
elsif env_key.start_with?("CONTENT_")
env_key.downcase.tr("_", "-")
end
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a divergence between agent mode and collector mode, in that the "Environment" section in agent mode contains other Rack CGI-flavoured attributes in it as well as HTTP headers, but in collector mode, only actual HTTP headers will be emitted as http.request.header.* attributes, and all other CGI-flavoured attributes in it from Rack will be dropped.

# stubs Excon). Verifies, end to end, that the inject-only middleware writes
# trace context onto a live outgoing request while the instrumentor's client
# event span is current -- the ordering the stubbed suite can't prove.
describe "Excon integration" do

@unflxw unflxw Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing Excon integration was previously untested -- this tests it just enough to assert that trace context propagation takes place.

@unflxw
unflxw requested review from lipskis and tombruijn July 8, 2026 17:42
@unflxw
unflxw marked this pull request as ready for review July 8, 2026 17:42
@tombruijn tombruijn added the feature A new feature for this component. label Jul 9, 2026
Comment on lines +763 to +782
def report_errors
if @backend.records_errors_eagerly?
run_error_blocks
else
report_errors_as_duplicates
end
end

# Eager mode: the errors are already recorded, so just run their blocks.
# Blocks run in add-order, so a later error's block wins on a shared key, and
# all block-set metadata merges onto the root span. (Per-error metadata
# isolation is deferred -- the processor/UI does not read per-event
# attributes yet.)
def run_error_blocks
@error_blocks.each_value do |blocks|
self.class.with_transaction(self) do
blocks.each { |block| block.call(self) }
end
end
end

@unflxw unflxw Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm realising now that this is likely not the right behaviour. We probably want to run error blocks immediately in collector mode. Unlike in agent mode, where we need to store them in order to fabricate duplicates at completion time, I see no benefit to storing them and running them later in collector mode.

I don't think error blocks should be used like this (in collector mode, they likely should not be used at all) but if the block contains things that should be attached to the current span, such as breadcrumbs (🌵), other errors (😨) or custom OpenTelemetry instrumentation (💀) then, by running them immediately, those would be attached to the right point in the span tree.

(to be clear: none of those are things that an error block should contain)

@unflxw unflxw Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c8fbb04.

The one non-obvious part is @error_blocks. Collector mode no longer stores the block there, but it still registers the error as a key, because we rely on those keys for last_errors, the distinct-error limit, and the duplicate check. And since blocks now run when added, report_errors only has work to do in agent mode, so run_error_blocks is gone.

Added a collector-mode test to lock it in: an error whose block adds a breadcrumb, asserting the breadcrumb lands on the event span, not the root. Fails without the change.

@backlog-helper

This comment has been minimized.

@unflxw
unflxw force-pushed the wip-otel-rework branch 3 times, most recently from 31d2771 to c220cce Compare July 13, 2026 12:41
@backlog-helper

This comment has been minimized.

@backlog-helper

This comment has been minimized.

1 similar comment
@backlog-helper

This comment has been minimized.

@unflxw
unflxw force-pushed the wip-otel-rework branch from e3eb9fb to 9a18f59 Compare July 16, 2026 11:39
unflxw added 6 commits July 16, 2026 16:49
Introduce an optional "collector mode": when `collector_endpoint` (or
`APPSIGNAL_COLLECTOR_ENDPOINT`) is set, `Appsignal.start` boots an
OpenTelemetry SDK that exports OTLP/HTTP to that endpoint alongside the
usual C-extension agent. No AppSignal-collected data routes through the
SDK yet — this commit stands up the mode, its configuration, and the
seams the later commits build on.

The OpenTelemetry gems stay optional (not gemspec dependencies): the
collector boot-checks for them and falls back to the agent when they are
absent or too old. Configuration gains the collector attribute options
(service name, filter/send/response settings) that shape the exported
resource.

Also adds the shared test infrastructure the rest of the work relies on:
the `-collector` gemfile variants, an in-memory OTLP mock server, and
the agent/collector dual-mode example helpers.
Route `Appsignal::Logger` through a backend seam: the extension backend
keeps the existing C-extension logging, and a new OpenTelemetry backend
emits log records through the SDK's logger provider when collector mode
is active. `Appsignal::Backends.logger` picks between them.

The OpenTelemetry backend maps Ruby severities and formats to their OTLP
equivalents and attaches `appsignal.group`/`appsignal.format` as hard
overrides so user attributes can't spoof them.
Route the custom metric helpers (`set_gauge`, `increment_counter`,
`add_distribution_value`) through `Appsignal::Backends.metrics`: the
extension backend keeps the C-extension path (including the out-of-range
warning), and a new OpenTelemetry backend records gauges, up/down
counters and histograms through the SDK's meter provider when collector
mode is active.

With both the metrics and logger backends in place, `Appsignal.start`
can fall back to the agent when the OpenTelemetry SDK fails to boot, so
telemetry is never silently dropped.
Dual-mode the GVL, MRI and Sidekiq probe specs so each metric example
runs in both agent and collector mode. The probes emit through the
metric helpers, so no probe code changes — this just proves their
output routes to OpenTelemetry when collector mode is active.
Back `Appsignal::Transaction` with a pluggable backend. The extension
backend keeps the existing C-extension behaviour (it now owns the
sample-data serialization, breadcrumb buffering and the agent's
error-cause shape). A new OpenTelemetry backend maps a transaction to an
OTel root span and each event to a child span, chosen by
`Appsignal::Backends.transaction`.

Because one `Transaction` drives two error models, the backend exposes
`records_errors_eagerly?`: the extension collects errors then duplicates
the transaction per error at completion, while the OTel backend records
each as an `exception` span event straight away.

Key mapping decisions, for reviewers:

- Span kind comes from the namespace (web/action_cable => SERVER,
  background_job => CONSUMER); datastore client events pass
  `opentelemetry_kind: :client`, set at creation since the kind is
  immutable.
- An event span's name is `title || name`, with the event name kept as
  `appsignal.category`; SQL bodies map to `db.query.text`, everything
  else to `appsignal.body`.
- Errors attach to the currently-open span with the cause chain in one
  `appsignal.error_causes` JSON attribute; a discarded transaction sets
  `appsignal.ignore_subtrace` so the collector drops it.
Dual-mode the hook, integration, Rack and loader specs so each existing
event-emission example runs in both agent and collector mode, asserting
the OpenTelemetry span shape (name, category, body, kind, parent) in
collector mode. No library changes — this exercises the trace backend
through every integration.

Trace-context propagation (incoming extraction, outgoing injection) is
left to the propagation commits, so the specs that also cover it are
dual-moded there alongside their library changes.
unflxw and others added 10 commits July 16, 2026 16:59
In collector mode, HTTP client integrations write the current W3C trace
context onto their outgoing requests so the downstream service continues
the same trace. Net::HTTP, HTTP.rb, Excon and Faraday each inject a
`traceparent` header (Excon via a dedicated middleware, HTTP.rb on every
hop), and mark the request span with CLIENT kind.

Injection is a no-op outside collector mode, so agent-mode behaviour is
unchanged. Excon moves to its own gemfile now that it is exercised
directly rather than only through Faraday's adapter.
In collector mode, the Rack middleware, Rack event handler and the
Webmachine integration read an incoming W3C `traceparent` header and
pass it to the transaction. A web (SERVER) transaction continues under
the remote span; a job (CONSUMER) transaction starts its own trace
linked back to it; with no context, or outside collector mode, it stays
a plain root span.

`Transaction.create` gains an `opentelemetry_context:` keyword that
threads through to the backend, which parents or links the root span
accordingly.
Carry the W3C trace context across the enqueue/perform boundary for
Sidekiq, Resque, Que, Shoryuken and Active Job. On enqueue (in collector
mode) each client writes the current context onto the job payload and
marks the enqueue span with PRODUCER kind; on perform the server side
reads it back and links the job's trace to the enqueuing span.

Each library carries the context on its own channel: a job hash key, an
SQS message attribute, a Que tag, or Active Job's serialized arguments.
It skips propagation rather than breaking the enqueue when a channel is
full, and is a no-op outside collector mode.
In collector mode each error is recorded when it is added, but its block
was still held until completion. Deferring only makes sense in agent
mode, where blocks run against the duplicate transactions fabricated at
completion. In collector mode there is no benefit to holding them back.

Run the block when the error is added instead. Side effects that target
the current span, such as breadcrumbs, nested errors or custom
instrumentation, then land where the error was reported rather than on
the root span at completion. None of those belong in an error block, but
this puts them in the right place if they are used.

With blocks no longer deferred in collector mode, the error tracking
that supported deferral was doing two jobs at once. `@error_blocks` was
both the set of distinct errors, which drives `last_errors`, the dedup
check and the error limit, and the per-error blocks that only agent mode
runs. Split them. `@errors` is now the distinct-error set, used in both
modes, and `@error_blocks` holds only blocks and is populated only in
agent mode. `@errors` is a Set, so membership is by object identity,
matching how the errors were deduplicated as hash keys before.

Finally, rename `records_errors_eagerly?` to
`supports_multiple_errors?`. The old name described the mechanism. The
real question at the call site is whether the backend can hold more than
one error on a transaction.
The sequel-rails gem emits its queries as `sql.sequel`
ActiveSupport::Notifications events. Those are recorded through the
generic notifications integration, not the dedicated Sequel hook, so
the hook's CLIENT-kind tagging never applied to them.

The notifications integration only listed `sql.active_record` as an
outgoing datastore call, so Sequel queries were exported with the
default (INTERNAL) span kind in collector mode. Add `sql.sequel` to
that list so a Sequel query is CLIENT regardless of which path
records it, matching ActiveRecord and the dedicated Sequel hook.
The `extract_rack_context` and `extract_job_context` helpers called
`OpenTelemetry.propagation.extract` without a `context:` argument, so it
defaulted to `Context.current`. The W3C extractor returns its base
context unchanged when the carrier has no `traceparent`, so a request
with no incoming trace context inherited whatever span happened to be
attached to the fiber.

In collector mode that let a span left over from an earlier request
become the parent of a later request's root span, merging unrelated
requests into a single trace. Extract onto `Context.empty` instead, so
extraction reflects only the carrier. A real incoming `traceparent`
still parents (web) or links (jobs) exactly as before.
A transaction runs blocks supplied by user code in a few places: the
error block passed to `set_error`, `send_error`, and `report_error`, and
the `after_create` and `before_complete` hooks. Any of them can raise.

These blocks can run far from where they were defined. In agent mode an
error block runs at completion, not when the error is added. So a raise
propagated out of whatever drove creation or completion, surfacing the
error in unrelated code. In collector mode it also skipped the backend's
completion step, which left the transaction's OpenTelemetry context
attached to the fiber. That leaked context then became the parent of the
next request's root span on the reused thread, merging unrelated
requests into one trace. Agent mode never showed the leak because its
extension handle is not a per-fiber stack.

Wrap each error block once, where user code hands it in, so it stays
protected wherever it later runs. `send_error` and `report_error` name
themselves when they wrap the block, so the log points the customer at
the call they made rather than at an internal method name. The
`after_create` and `before_complete` hooks are guarded where they run,
because they can be registered with a block or by pushing a method onto
the set, so there is no single entry point to wrap them at.

On failure, log the error and the block's definition site, then carry on
so creation and completion always finish. The error is swallowed rather
than re-raised, because the block can run far from its caller and
re-raising would surface it in code that has nothing to do with it.
A web request that never sets an action name (for example a static
asset served without a controller) has no action to group by. Agent
mode does not report such a transaction. In collector mode the root
span is created with a placeholder name, `appsignal.transaction
<namespace>`, so every actionless request was surfacing under that
one shared placeholder action.

Track whether an action was set and, when it was not, flag the root
span with `appsignal.ignore_subtrace` on completion, exactly as
`discard` does. The AppSignal Collector (0.10.0 and newer) then drops
the subtrace, so these requests no longer appear, matching agent mode.
Enqueuing a Delayed Job now records an `enqueue.delayed_job` event on
the active transaction, so enqueues made from a web request or another
job show up in the event timeline. Delayed Job was the only background
job backend still missing this.

In collector mode that event also opens a producer span, matching the
other job backends and OpenTelemetry's own Delayed Job instrumentation.
Delayed Job has no envelope to carry trace context across the
enqueue/perform boundary, so -- like OpenTelemetry -- nothing is
injected and the producer and consumer spans are not linked.

Test it against Delayed Job's in-memory test backend under a new
`delayed_job` gemfile, driving the real enqueue and perform lifecycle in
both agent and collector mode.
Audit of the transaction and helper specs for behaviors covered in only
one mode, agent or collector, that should be covered in both. Most gaps
were agent-only leftovers that assert through `to_h`-based matchers,
which do nothing in collector mode.

Fold the two send_error block specs into one dual-moded spec. It sets an
active transaction, calls `send_error` with a block, and checks the
block's metadata and the error land on `send_error`'s own transaction
while the active one is restored and left untouched. In collector mode
this confirms the active trace continues rather than being dropped or
left open.

Cover more behaviors in both modes: the `set_error` and `report_error`
blocks, `set_empty_params!`, the `add_params`, `add_session_data`,
`add_headers` and `add_custom_data` merge and block variants, `set_action`
and `set_namespace` setting the value, `report_error` not completing the
active transaction, and a transaction that reports a cause-less error
after one with causes.

Also add the one missing collector metric-coercion case, a symbol metric
name, to the OpenTelemetry metrics backend spec.
@unflxw
unflxw force-pushed the wip-otel-rework branch from e03b091 to 1375eeb Compare July 16, 2026 15:18
@backlog-helper

Copy link
Copy Markdown

This is a message from the daily scheduled checks.

New issue guide | Backlog management | Rules | Feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature A new feature for this component.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants