Skip to content

Add Linux user_events Receiver#2717

Draft
lalitb wants to merge 56 commits intoopen-telemetry:mainfrom
lalitb:user-events-receiver
Draft

Add Linux user_events Receiver#2717
lalitb wants to merge 56 commits intoopen-telemetry:mainfrom
lalitb:user-events-receiver

Conversation

@lalitb
Copy link
Copy Markdown
Member

@lalitb lalitb commented Apr 20, 2026

Change Summary

Adds a contrib receiver for ingesting events through Linux user_events tracepoint.
The receiver reads configured Linux user_events tracepoints through per-CPU perf sessions, decodes supported log payloads, and emits OTAP log records into the pipeline. It includes batching, late tracepoint registration, downstream-full overflow handling, memory-pressure ingress shedding, drop/loss metrics, and Linux-only feature gating.

Detailed configuration and runtime behavior are documented in-
crates/contrib-nodes/src/receivers/userevents_receiver/README.md.

What issue does this PR close?

How are these changes tested?

Are there any user-facing changes?

lalitb added 30 commits April 16, 2026 19:16
The Linux userevents receiver branch could register the producer tracepoint but failed to open and drain a usable perf session on the Azure VM. This change simplifies perf session setup, surfaces actionable session-open errors, lowers the demo wakeup watermark for tiny traffic, and adds a repeatable VM demo script so the exporter-to-receiver-to-debug path can be reproduced.

Constraint: Azure VM user_events perf sessions rejected the earlier clockid-based setup with EINVAL
Constraint: Tiny demo traffic never crossed the default 256 KiB wakeup watermark
Rejected: Keep the default wakeup watermark and wait longer | the drain loop never woke for the small demo payloads
Rejected: Leave perf ID discovery on read() | session setup should not depend on sample reads
Confidence: medium
Scope-risk: moderate
Reversibility: clean
Directive: For single-core demos, pin the producer to the receiver CPU or widen CPU coverage explicitly
Tested: cargo check -p otap-df-core-nodes
Tested: Azure VM end-to-end demo with exporter pinned to CPU 0 and debug output written to userevents-debug-output.log
Not-tested: cargo xtask check (fails on existing branch issues in arrow_records_encoder.rs and clippy outside this commit)
Not-tested: Multi-core userevents ingestion behavior
The user-events receiver is Linux-specific and not part of the standard OpenTelemetry receiver set, so it belongs in contrib rather than core. This change relocates the receiver module and registration path, replaces the OTEL SDK-based Linux E2E producer with a tiny standalone EventHeader emitter, and removes stale demo/assessment files while keeping the VM demo path reproducible.

Constraint: The Azure VM demo only drained low-volume user_events traffic once the wakeup watermark was reduced and the producer was pinned to the receiver CPU
Constraint: The workspace forbids unsafe test code, so CPU pinning in the standalone emitter had to use a safe crate
Rejected: Keep the receiver in core-nodes and only re-export it from contrib | that would preserve the wrong ownership boundary
Rejected: Keep the OTEL SDK and opentelemetry-user-events-logs dev-dependencies for the Linux E2E test | the receiver only needs an EventHeader producer with the accepted Common Schema field layout
Confidence: medium
Scope-risk: moderate
Reversibility: clean
Directive: If you run the single-core VM demo, keep the producer pinned to CPU 0 or broaden the receiver CPU coverage
Tested: cargo check -p otap-df-contrib-nodes
Tested: cargo check -p otap-df-otap
Tested: cargo check -p otap-df
Tested: Azure VM Linux E2E test cargo test -p otap-df-otap --test userevents_exporter_receiver_e2e -- --ignored --nocapture
Not-tested: cargo xtask check (blocked by existing branch issues outside this commit)
Not-tested: Multi-core userevents receiver behavior after the move
The original VM demo proved the user-events path on a single CPU, but that was not enough to explain how the real OTEL exporter behaves when its caller thread can run on any core. This change lets the demo run df_engine across a chosen CPU set, launches one pinned producer process per CPU, and summarizes the CPUs observed by the receiver so the all-core story is visible in one run.

Constraint: The real OTel user-events exporter writes on the caller thread and does not create per-CPU sender threads
Constraint: Single-core demos are deterministic only when the producer is pinned to the receiver CPU
Rejected: Keep the demo single-core and explain multi-core behavior only in prose | that leaves the most important runtime property unverified
Rejected: Add producer threading changes inside the OTEL exporter example | the demo only needs orchestration around existing binaries
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Use DEMO_CORES=all for broad coverage and a small integer like 2 or 4 for readable live demos
Tested: bash -n scripts/run-userevents-azure-vm-demo.sh
Tested: Azure VM end-to-end demo with DEMO_CORES=2 showing linux.userevents.cpu=0 and linux.userevents.cpu=1 in debug output
Not-tested: DEMO_CORES=all on the full 64-core VM for extended durations
The receiver was writing raw perf-clock timestamps into OTLP time_unix_nano, which made Geneva show 1970-era event times. On the test VM, asking perf for CLOCK_REALTIME timestamps was rejected with EINVAL, so this change converts monotonic perf sample times to wall clock in-process and preserves the working VM path.

The same VM also exposed a second issue: tracefs event metadata files such as user_events/.../id are root-readable only. The receiver was collapsing that EACCES into a fake MissingTracepoint retry loop, which made permission problems look like late registration. This now reports the permission error directly so operators know they need elevated privileges or relaxed tracefs reads.

Constraint: VM perf_event_open rejected use_clockid/CLOCK_REALTIME with EINVAL
Constraint: tracefs user_events metadata files are root-readable only on the demo VM
Rejected: Force perf CLOCK_REALTIME timestamps | broke session open on the target VM
Rejected: Treat unreadable tracefs metadata as missing tracepoints | hid the actual permission failure behind infinite retry logs
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: If this receiver later moves onto a session abstraction like mdsd/TracepointSession, replace the local monotonic-to-realtime conversion with session-backed time conversion
Tested: cargo check -p otap-df-contrib-nodes --features geneva-exporter
Tested: VM detailed timestamp verification showed wall-clock Timestamp/ObservedTimestamp values after rebuild
Tested: VM sudo df_engine + basic-logs reached userevents_receiver.session_opened and geneva_exporter.upload
Not-tested: Non-sudo df_engine with tracefs permissions explicitly relaxed for azureuser
The userevents receiver now routes low-level Linux collection through a shared one_collect-backed adapter while keeping receiver-specific decode and OTAP output in this repo.

This creates the boundary we want for future platform expansion: collection/session code can be shared with a later ETW receiver, while decode and output remain platform-specific above that line. The integration is target-gated so macOS host builds do not pull one_collect, and Linux validation was completed against the VM checkout.

Constraint: one_collect is not published as a normal dependency, so it is brought in as a vendored submodule
Constraint: the crate currently requires its scripting feature to compile correctly in dependency mode on Linux
Rejected: Keep extending the bespoke userevents session without a shared backend | makes future ETW collection reuse harder
Rejected: Copy one_collect internals into this repo | creates long-term divergence from the actively maintained upstream
Confidence: medium
Scope-risk: moderate
Reversibility: clean
Directive: Keep one_collect isolated behind the collection adapter; do not leak one_collect session/event types into receiver decode/output layers
Tested: cargo check -p otap-df-contrib-nodes --features geneva-exporter (macOS host)
Tested: cargo check -p otap-df-contrib-nodes --features geneva-exporter on /home/azureuser/otel-arrow/rust/otap-dataflow (Linux VM)
Not-tested: End-to-end VM runtime with the one_collect-backed session replacing the previous session implementation
The first one_collect-backed userevents adapter used callback-time wall clock to avoid the crate-level unsafe-code ban, but that changed the semantics of the emitted timestamp. This follow-up restores record-time semantics by reading the sample timestamp through one_collect's safe time_data_ref path and aligning it to wall clock with the library's qpc helper instead of direct libc clock syscalls.

Constraint: contrib-nodes is built with -D unsafe-code
Constraint: the adapter must preserve event-time semantics suitable for Geneva while remaining within one_collect's safe API surface
Rejected: Keep callback-time SystemTime::now() timestamps | drifted from mdsd-style record timestamp semantics
Rejected: Use direct clock_gettime via unsafe libc calls | violated the crate's unsafe-code policy
Confidence: medium
Scope-risk: narrow
Reversibility: clean
Directive: If one_collect later exposes an explicit safe realtime conversion for perf sample timestamps, prefer that over the current qpc-to-wall-clock alignment logic
Tested: cargo check -p otap-df-contrib-nodes --features geneva-exporter (macOS host)
Tested: cargo check -p otap-df-contrib-nodes --features geneva-exporter on /home/azureuser/otel-arrow/rust/otap-dataflow (Linux VM)
Not-tested: End-to-end VM Geneva runtime after this exact timestamp-path adjustment
The Linux-only userevents smoke test and standalone Common Schema emitter now live under contrib-nodes, alongside the receiver implementation they exercise. The move also carries the minimum dev-dependency wiring and test-shape fixes needed to keep contrib-nodes green under the full workspace check after the branch's one_collect work.

Constraint: receiver-specific tests should follow the contrib-nodes ownership boundary
Rejected: Keep the tests under otap | obscures ownership after moving the receiver implementation
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep receiver-specific integration tests with contrib-nodes unless they require shared otap-only harness code
Tested: cargo check -p otap-df-contrib-nodes; cargo test -p otap-df-contrib-nodes --test userevents_exporter_receiver_e2e --no-run; cargo clippy -p otap-df-contrib-nodes --all-targets -- -D warnings; cargo xtask check
Not-tested: ignored Linux kernel user_events runtime execution for the moved integration test on host
The one_collect-backed userevents receiver now lives and tests inside contrib-nodes, so core-nodes and otap no longer need to carry receiver-only dependency baggage. This keeps ownership boundaries honest and avoids implying cross-crate coupling that no longer exists.

Constraint: The receiver implementation and its Linux-only test harness now live in contrib-nodes
Rejected: Leave the extra deps in place | obscures ownership and invites future accidental coupling
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep userevents-specific dependencies scoped to contrib-nodes unless another crate gains direct Rust usage
Tested: cargo xtask check (clippy passed with these removals; workspace test failure remained the pre-existing fake_data_generator network clone issue); dependency usage re-scan in core-nodes and otap
Not-tested: A fully clean offline-free workspace test run, because fake_data_generator still depends on runtime network access to semantic-conventions
Related: db2ec76
Related: 5af9bbb
Related: a08f20e
The checked-in Azure VM and Multipass helpers were environment-specific orchestration rather than reusable project assets, and one of them had already gone stale after the userevents test moved into contrib-nodes. The remaining checked-in documentation now points at the real contrib-nodes test location instead of those ad hoc scripts.

Constraint: Scripts committed to the repo should be generally usable or clearly owned project tooling
Rejected: Keep the scripts and update them | still leaves personal/demo-specific orchestration in the tree
Rejected: Move the scripts into contrib-nodes | does not fix that they are host/VM-specific rather than crate assets
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep user-specific VM/demo automation out of the repo; document reproducible manual steps instead
Tested: npx --yes markdownlint-cli crates/contrib-nodes/src/receivers/userevents_receiver/README.md; python3 /Users/lalitb/work/obs/otel/rust/lalitb/otel-arrow/tools/sanitycheck.py
Not-tested: Runtime execution of the deleted scripts, because the cleanup intentionally removes rather than preserves them
Related: db2ec76
Related: 9b05b3e
[target.'cfg(target_os = "linux")'.dependencies]
# TODO: Remove this pinned git dependency once one-collect is upstream in the
# normal dependency graph and no longer needs to be pulled directly here.
one_collect = { git = "https://github.com/microsoft/one-collect.git", rev = "2d33e674348168f4a19bd3ab40d92f1c556407ec", features = ["scripting"], optional = true }
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: put the date of the commit here in comment, so it is to tell which date's version we are consuming here.


- Reads Linux `user_events` tracepoints through per-CPU perf sessions
- Supports single-tracepoint and multi-tracepoint configuration
- Supports the `common_schema_otel_logs` decode format for supported log payloads
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Supports the `common_schema_otel_logs` decode format for supported log payloads
- Supports the `microsoft_common_schema_logs` decode format for supported log payloads
  1. I think it's best to quality CS with Microsoft, as it is hard for users to known what is "common" about it.
    2.CommonLogs is just CS Logs - its inspired by OTel Logs data model, but not tied to it, so I suggest avoiding the term OTel Logs, Just "Logs" is good.

through `perf_event_open` and converts them into OTAP logs for downstream
processing.

> **Note:** The current decoder supports the Microsoft Common Schema OTLP logs
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> **Note:** The current decoder supports the Microsoft Common Schema OTLP logs
> **Note:** The current decoder supports the Microsoft Common Schema logs


`user_events` writes enter Linux tracing as user-space tracepoint events. The
kernel stores trace records in per-CPU ring buffers; each receiver instance
drains the ring for the CPU/pipeline core it owns, decodes the EventHeader
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

out of curiosity-what if 2 instances of df_engine is run on a node - will they both read the same events from the ring buffer? Nothing wrong with that, but making sure we anticipate such scenarios and make no assumptions otherwise.

**Placement rule:** a pipeline assigned to core `K` opens the perf ring for
CPU `K` only. If an application thread writes a `user_events` record while the
kernel has scheduled it on CPU `X`, that record lands in the CPU `X` ring and
is only visible to the receiver instance that owns CPU `X`. No other receiver
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

receiver instance that owns CPU 'x" feels incorrect... Receiver runs on CPU X, and it only reads from that CPU X. It cannot assume ownership of ring buffer X as another df_engine instance could also run.

v
receiver pipeline pinned to CPU K

No other receiver pipeline reads CPU K's ring.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
No other receiver pipeline reads CPU K's ring.
No other receiver pipeline in this df_ngine instance reads CPU K's ring.

No other receiver pipeline reads CPU K's ring.
```

For an 8-core host, coverage looks like this:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: user 4 core example to make it slightly easier to read.

this receiver deployment. They are not seen and then discarded; no receiver is
attached to that CPU's ring. The engine does not emit a metric for these
missing records because they are never observed by the receiver; they are lost
silently.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do mention that silently refers to silent from df_engine standpoint. The sender could very well check is_listener_available() and use that to emit its own metric, so it won't be totally silent loss... Maybe too much details? Won't hurt!


| Configuration | Writer pinned? | Result |
|----------------------------------|----------------|------------------------------------------------------------|
| `--num-cores 1` | No | Racy; writes on CPUs 1..N not collected by this receiver |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does "Racy" means?

Current implementation supports:

- single-tracepoint and multi-tracepoint configuration
- `common_schema_otel_logs`, which decodes Microsoft Common Schema OTLP logs,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `common_schema_otel_logs`, which decodes Microsoft Common Schema OTLP logs,
- `common_schema_otel_logs`, which decodes Microsoft Common Schema logs,

batching:
max_size: 512
max_duration: 50ms
overflow:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does other receivers has such a setting to define overflow behavior?

max_bytes_per_turn: 1048576
max_drain_ns: 2ms # duration string: 2ms, 500us, 1000000ns
batching:
max_size: 512
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

512 is count of items? or size? the config is not clear...

per_cpu_buffer_size: 1048576 # bytes
```

Exactly one of `tracepoint` or `subscriptions` must be configured.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the example has tracepoint as a sub-setting within subscriptions.. so not sure what this statement is referring to . Is that a stale comment?

| Field | Default | Description |
| --- | --- | --- |
| `tracepoint` | none | Single tracepoint shorthand. Must use `user_events:<event>`. Mutually exclusive with `subscriptions`. |
| `subscriptions` | none | List of tracepoints. Each entry must use `user_events:<event>` and `format.type: common_schema_otel_logs`. |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we avoid subscriptions concept, and just make tracepoint config a [], so users can put 1 or more tracepoints there? Sorry I am missing why we have subscription concept.

| `tracepoint` | none | Single tracepoint shorthand. Must use `user_events:<event>`. Mutually exclusive with `subscriptions`. |
| `subscriptions` | none | List of tracepoints. Each entry must use `user_events:<event>` and `format.type: common_schema_otel_logs`. |
| `format.type` | `common_schema_otel_logs` | Only supported decode format. |
| `session.per_cpu_buffer_size` | `1048576` | Requested per-CPU perf ring size in bytes. Rounded by the underlying perf/ring setup. |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is it common to specify size in bytes? If yes, do we support human friendly settings like 10KB?

| `format.type` | `common_schema_otel_logs` | Only supported decode format. |
| `session.per_cpu_buffer_size` | `1048576` | Requested per-CPU perf ring size in bytes. Rounded by the underlying perf/ring setup. |
| `session.wakeup_watermark` | `262144` | Reserved for future one_collect wakeup support; currently ignored. |
| `session.late_registration.enabled` | `false` | When true, keep retrying if the tracepoint is not registered yet. |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some explanation when this is relevant. like the engine starts up first, and apps start much later, so engine won't find the tracepoints when it starts and hence it has to keep poling.

(or is there an alternate where the engine just creates the tracepoint itself..?)

| `drain.max_records_per_turn` | `1024` | Maximum records popped from the receiver's pending queue per drain turn. |
| `drain.max_bytes_per_turn` | `1048576` | Maximum payload bytes popped per drain turn. |
| `drain.max_drain_ns` | `2ms` | Total drain-turn budget. Accepts duration strings such as `2ms`, `500us`, or `1000000ns`; must be greater than zero. |
| `batching.max_size` | `512` | Flush once this many logs are buffered in the current Arrow batch. |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a note about why this batching is required. (something like sending one arrow batch with 1 record is inefficient etc..)

receiver uses `level` only as a fallback when the payload does not provide
`severityNumber`:

| EventHeader level | Fallback severity |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels an arbitrary mapping..lets link to OTel Log data model, so this makes clear sense.

| PartB.severityNumber | `severity_number` |
| PartB.severityText | `severity_text` |
| PartB.body | log `body` |
| PartB.eventId | typed `eventId` attribute |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can drop this thing... OTel Rust producer also need to drop this. EventName is serving the needs currently, adding a numerical id, specially treated inside attribute is additional complexity nobody outside us may understand!

| PartB.severityText | `severity_text` |
| PartB.body | log `body` |
| PartB.eventId | typed `eventId` attribute |
| PartC scalar fields | flat log attributes with original field names and basic value types |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO for supporting non-basic? or we can make a statement that user_events only support basic types, and producers are expected to produce that only?

bytes, `__csver__` missing or not first, `__csver__ != 0x400`, missing PartB,
PartB `_typeName != "Log"`, duplicate PartA/PartB/PartC structs, unknown PartB
fields, or invalid nesting. On failure, the receiver emits the raw payload as a
base64 body and increments `cs_decode_fallbacks`.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

fields and flat application attributes; receiver transport/debug metadata is
not emitted as log attributes.

The only PartB field emitted as a log attribute is:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as suggestd elsewhere, its probably best to remove this special casing.

- `body`
- `severity_number` / `severity_text`
- `event_name` (prefers PartB.name, falls back to EH.Name / PartA.name)
- `time_unix_nano` (from PartA.time when present, else the perf sample timestamp)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't;t we also check for observed_ts, before falling back to perf sample timestamp?

transport/diagnostic fields such as tracepoint name, provider name,
EventHeader level/keyword, CPU, PID/TID, sample id, payload size, body
encoding, or decode mode. These describe the receiver itself rather than the
application payload; surfacing them as OTLP log attributes would pollute
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pid/tid are quite useful, and describes the app itself right? worth considering, but not crucial for v1.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pid is important I think, as multiple processes can exist and this is only way to distinguish?
anyway, not blocker for the 1st PR.

- permission to read tracefs metadata
- permission to use `perf_event_open` for the configured tracepoints

`user_events` was merged in Linux 6.4; earlier kernels do not expose
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice. I think I left a comment about documenting this elsewhere, but just saw you already had that here!

}
}

#[ignore = "requires Linux kernel user_events support and write permissions to tracing/user_events_data"]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't we have that in the Github Linux runners? We run user_events tests in CI in OTel Rust Contrib.

Comment on lines +63 to +66
/// Keep the receiver identity vendor-neutral. The current implementation only
/// supports the Common Schema OTLP logs decoder, but the receiver is intended
/// to support pluggable schema mappings in the future.
pub const USEREVENTS_RECEIVER_URN: &str = "urn:otel:receiver:userevents";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should aim for a generic user_events receiver rather than one that bakes in Common Schema mapping. A few thoughts on how this could be structured:

1. Start with what the Linux ecosystem already supports.

A generic user_events receiver should, at minimum, decode events the same way standard Linux tools (perf, ftrace, etc.) do (using the tracefs format string declared at registration time). If a producer registers "my_event u32 status; char[] endpoint", the receiver should emit those as typed attributes on a log record. The tracepoint_decode crate already supports this via its TraceEventFormat decoding style. I don't think in its current state, the PR handles this path. It only decodes EventHeader payloads.

2. Extend with EventHeader decoding as an opt-in feature.

tracepoint_decode auto-detects whether a tracepoint uses standard format-string fields or the EventHeader self-describing convention (by checking if the first field is named eventheader_flags). We could support both decode paths in the receiver, with EventHeader gated behind a feature flag. For standard events, attributes come from the format-string fields. For EventHeader events, attributes come from the self-describing metadata. Both land as typed attributes on OTAP/OTLP log records. There is no semantic interpretation at the receiver level.

This gives us a genuinely generic receiver that works with any user_events event. The EventHeader feature flag lets us keep that path opt-in since EventHeader is a custom userspace convention.

3. Model the Common Schema mapping as a processor.

The CS-specific logic (checking __csver__ == 0x400, promoting PartB.bodyLogRecord.body, parsing PartA.ext_dt_traceId → typed trace ID, mapping ext_cloud_roleservice.name, etc.) is essentially attribute transformation: promote, rename, delete. This fits naturally as a processor in contrib-nodes. The receiver emits decoded EventHeader fields as attributes; the CS processor promotes the known ones to OTAP/OTLP typed fields and strips the originals.

This separation would allow for a generic receiver with a vendor-neutral name and scope, while the custom Common Schema interpretation can reside in a contrib processor node. We could even consider making user_events reciever a core node at that point.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. Start with what the Linux ecosystem already supports.

+100

2. Extend with EventHeader decoding as an opt-in feature.

+1

3. Model the Common Schema mapping as a processor.

Not sure, I guess there are many ways to do this. Having said that, I think it'll be great to focus on building a great user_events receiver first, and experiment with it to see what can be done to fit the Microsoft need (extension, etc.). I don't think anyone here would want to build a user_events receiver based on Microsoft's specific need.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense; I’ll align the PR around a generic Linux user_events receiver first, with tracefs-format decoding as the baseline and EventHeader handled as an explicit extension point, while keeping Microsoft/CommonSchema-specific mapping outside the receiver boundary.

@lalitb lalitb marked this pull request as draft April 24, 2026 23:30
@lalitb
Copy link
Copy Markdown
Member Author

lalitb commented Apr 24, 2026

Moving to draft for refactor as discussed here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Proposal: Add Linux user_events Receiver for Telemetry from Applications Co-Located with the Collector

5 participants