Cross service trace propagation#375
Merged
Merged
Conversation
Collector mode adds W3C trace-context propagation, but no setup showed a trace spanning two AppSignal apps: the HTTP client actions called localhost (the app calling itself), and background jobs ran under a single app name. Make rails7-sidekiq collector-capable and run a second instance of the same image as a distinct "downstream" app. The HTTP client actions now call it instead of themselves, and a new DownstreamWorker runs on a queue only the downstream service drains, alongside the existing same-service worker. In collector mode the trace continues across both apps over HTTP, and the job's trace links back to the enqueue span. Exercises appsignal-ruby#1535.
Add a second instance of the image as a distinct "downstream" app that drains the `downstream` Resque queue this app enqueues to. In collector mode the job's trace links back to the enqueue span across both apps. The existing same-service worker (default queue) is unchanged. Also stop instrumenting the Resque worker as a rake task (enable_rake_performance_instrumentation = false). Running `rake resque:work` with that on opens a long-lived "rake" transaction that every job reuses, so jobs were recorded under the `rake` namespace instead of `background_job` -- in agent mode too, since rake instrumentation was added. That also blocks the collector-mode enqueue->perform link (the perform span is typed Server, not Consumer). Exercises appsignal-ruby#1535.
Add a second instance of the image as a distinct "downstream" app that drains the `downstream` Que queue this app enqueues to (via --queue-name). In collector mode the job's trace links back to the enqueue span across both apps. The existing same-service worker (default queue) is unchanged. Exercises appsignal-ruby#1535.
Add a second instance of the image as a distinct "downstream" app that drains the `downstream` SQS queue this app enqueues to. In collector mode the trace context rides an SQS message attribute, so the job's trace links back to the enqueue span across both apps. The existing same-service worker (default/native queues) is unchanged. The `downstream` queue is created in the mock SQS server by both services (idempotent) so it exists regardless of boot order. Exercises appsignal-ruby#1535.
Add a second instance of the image on port 4002 as a distinct "downstream" app, and a call_downstream resource that makes an HTTP request to it. In collector mode the Net::HTTP integration injects the trace context and the downstream app extracts it through Webmachine's own (non-Rack) extraction path, so the trace spans both apps. This is the one HTTP-server extraction path not shared with the Rack integrations. Exercises appsignal-ruby#1535.
In collector mode both the AppSignal Ruby gem and Python package read APPSIGNAL_SERVICE_NAME and send it as the OpenTelemetry `service.name` resource attribute. AppSignal groups traces into a `<service>/<namespace>` namespace based on it. When the variable is unset the service name falls back to "unknown". Set it explicitly to "app" for every collector setup so none of them report an "unknown" service. The cross-service setups additionally name their second service "downstream", which sets up the next change: sharing a single app name across the two services in collector mode.
The cross-service setups run a second `downstream` service to exercise trace propagation between two separately-instrumented processes. In agent mode those two must stay separate AppSignal apps, because the agent has no service name concept and an app is the only identity it reports. Collector mode does have a service name, so model the pair the way it really is: one application made of two services. Point both services at the same APPSIGNAL_APP_NAME and let APPSIGNAL_SERVICE_NAME (app and downstream) tell them apart. The trace then propagates across services within a single app instead of spanning two apps.
In every Ruby setup that runs background jobs, the worker ran as a second process inside the `app` container, next to the web server. Move it into a dedicated `worker` container that runs the same image. This mirrors a real web/worker deployment, and it attributes job traces to a separate identity from the web requests. In agent mode the worker reports to a `<name>-worker` app. In collector mode it shares the app name and sets `APPSIGNAL_SERVICE_NAME=worker`, so its jobs group under a `worker` service. The `worker` service is image-only; only `app` declares `build:`. When two services build the same image tag, `docker compose build` fails on Docker's containerd image store with an "image already exists" error. For the same reason the existing `downstream` services are now image-only too. Bare `ruby/shoryuken` is left out because it has no agent or collector mode files to carry the worker's identity.
A DB-backed queue worker (que, solid_queue, delayed_job, good_job) polls a table that a migration creates. run-worker only waited for gems, so the worker could start before the app finished migrating, crash on the missing table, and stay dead, because processmon restarts a process on file changes but not when it exits. Wait for the app to serve HTTP before starting the worker. The app only starts its web server after it migrates, so a reachable app means the schema is ready. Redis-backed workers (sidekiq, resque) did not hit this, but the wait is harmless and keeps every run-worker identical.
The Django apps' make_request view called an external, uninstrumented URL (appsignal.com), so nothing continued the trace. Point it at a second, separately-instrumented instance of the same app (the `downstream` service) via DOWNSTREAM_URL, so the trace spans both. Falls back to the external URL when unset, so single-service runs still work. In agent mode the downstream is a separate `-downstream` app. In collector mode it shares the app name and sets APPSIGNAL_SERVICE_NAME=downstream, so it is a distinct service. Python deps install per container, not into a shared volume. The downstream container waits until the app serves HTTP before installing. That one signal means the app has built the appsignal dist into the shared /integration volume (so we install from it instead of racing its `hatch build`) and has migrated (so we never migrate from here). `downstream` is added to ALLOWED_HOSTS, since Django rejects the `Host: downstream:4002` header otherwise. The service is image-only so `docker compose build` does not collide on the shared image tag. Only the Django apps are changed, because they are the only Python setups that make an outbound HTTP request.
The Django apps ran their Celery worker, and flask-pika ran its RabbitMQ consumer, as a second process inside the `app` container next to the web server. Move it into a dedicated `worker` container that runs the same image, mirroring a real web/worker deployment. In agent mode the worker reports to a `<name>-worker` app. In collector mode it shares the app name and sets APPSIGNAL_SERVICE_NAME=worker, so its work groups under a `worker` service. The worker container waits for the app to serve HTTP before installing, the same readiness signal the downstream service uses: it means the appsignal dist is built in the shared /integration volume and the schema is migrated. The service is image-only so `docker compose build` does not collide on the shared image tag.
The downstream service runs a second web server and, for the DB-backed setups, a queue worker (Que) that polls a table a migration creates. Like run-worker before this, run-downstream only waited for gems, so it could start before the app migrated, crash, and stay dead, because processmon restarts a process on file changes but not when it exits. Wait for the app to serve HTTP before starting downstream. The app only starts its web server after it migrates, so a reachable app means the schema is ready. This matches the run-worker fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements changes in Ruby and Python test setups to enable to test cross-service trace propagation as implemented in the AppSignal platform in general, and in appsignal/appsignal-ruby#1535 specifically.
For HTTP requests, boot up the web application's container twice, as
appand asdownstream. Have the server atappperform HTTP requests against the server atdownstream, exercising both the trace context injection in the HTTP client's end and the extraction in the HTTP server's end.For background jobs, boot up a different container as
workerto process the jobs.When
mode=collector, use the same app name for all containers, with a different service name for each. Whenmode=agent, use different app names. Trace propagation only works in collector mode.