Skip to content

Request tracing support across core and connectors - #489

Merged
DImuthuUpe merged 22 commits into
masterfrom
tracing-impl
Jun 12, 2026
Merged

Request tracing support across core and connectors#489
DImuthuUpe merged 22 commits into
masterfrom
tracing-impl

Conversation

@lahirujayathilake

@lahirujayathilake lahirujayathilake commented Jun 5, 2026

Copy link
Copy Markdown
Member

Summary

Adds a single trace ID to every audit row a request produces. The trace ID flows through the core service, the event bus, and the AMIE, COmanage, and SLURM connectors. Also adds a small set of admin endpoints under /audit/* so operators can look up a trace, see the full event chain, and filter by subsystem, status, or time window.

Before this change, when a multi-step request failed somewhere (for example, an AMIE provisioning request that triggered downstream COmanage work), audit rows existed but had no link between them. There was no way to figure out "what was the full chain of events for this request, and where did it break?". This PR fixes that.

How it works

Audit rows are written by the components that do the work. AMIE writes structured rows as it walks each incoming packet through its state machine. The COmanage and SLURM connectors write rows when their subscribers start handling an event, and again on every success or failure outcome. Every row carries a trace ID, a span ID, a parent span ID, and a source label (core, amie, comanage, or slurm). The IDs let the admin endpoints rebuild the event tree. The source label is used for filtering.

Every audit row in Custos lives in one audit_events table. Core, COmanage, SLURM, and AMIE all write to it with the same shape, tagged by source. AMIE used to keep its own amie_audit_log table with foreign keys into AMIE-specific tables. That is gone in this PR. The AMIE-specific references it still needs (the originating packet and the per-attempt processing event) now live in a small table called amie_audit_extras, joined to audit_events on audit_event_id. The trace viewing layer reads audit_events directly, no UNION, and the source label tells operators which subsystem produced each row.

The shape generalizes. Any future connector that needs to attach its own references to an audit row creates a <connector>_audit_extras table and a matching /connectors/{name}/... endpoint that joins it. For now there is one connector-specific endpoint, GET /connectors/amie/packets/{packet_id}/audits, which returns every audit row written for one AMIE packet. The core audit_events table stays neutral and never grows connector-shaped columns.

The admin endpoints are read-only. They let you list traces with filters, fetch the event tree for one trace, fetch a flat time-ordered list for a trace or a single span, and list the available subsystems for the filter UI.

@lahirujayathilake
lahirujayathilake marked this pull request as ready for review June 5, 2026 21:58

@DImuthuUpe DImuthuUpe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to closely look at the data models again the verify whether we can compress those further

Comment thread pkg/models/trace.go Outdated
Comment thread pkg/models/trace.go
Comment thread pkg/models/trace.go
Comment thread pkg/models/trace.go
@lahirujayathilake
lahirujayathilake marked this pull request as draft June 8, 2026 20:47
@lahirujayathilake
lahirujayathilake marked this pull request as ready for review June 10, 2026 08:03
@lahirujayathilake
lahirujayathilake marked this pull request as draft June 10, 2026 20:48
@lahirujayathilake
lahirujayathilake marked this pull request as ready for review June 10, 2026 21:26
@DImuthuUpe

Copy link
Copy Markdown
Contributor

@lahirujayathilake please fix merge conflicts

@lahirujayathilake

Copy link
Copy Markdown
Member Author

@lahirujayathilake please fix merge conflicts

@DImuthuUpe, I updated the PR

@DImuthuUpe
DImuthuUpe merged commit 4759f2e into master Jun 12, 2026
1 check passed
@lahirujayathilake
lahirujayathilake deleted the tracing-impl branch June 25, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants