[Enhancement]: Add daemonless local ATOF sinks for subprocess stdin and inherited file descriptors

[smallstepman]

Affected area

Observability or exporters

Problem or opportunity

NeMo Relay currently supports ATOF observability output through filesystem and network-oriented sinks, but there is no built-in way to stream ATOF directly to another local process or to an inherited file descriptor.

This makes daemonless local integrations harder than necessary.

A harness that already embeds NeMo Relay may want to stream its ATOF event stream directly into:

• another binary's stdin,
• an inherited pipe or file descriptor provided by a parent process,
• stdout/stderr,
• or another local IPC mechanism,

without:

• writing temporary files,
• opening a TCP/HTTP/WebSocket endpoint,
• operating a collector or sidecar,
• or implementing a custom Relay plugin/subscriber solely for local output.

Two complementary primitives would cover these use cases well:

1. A subprocess sink where Relay launches a consumer binary and writes ATOF to its stdin.
2. An inherited file-descriptor sink where the caller owns the process topology and Relay writes to an already-open descriptor.

For the simple case:

Relay-enabled harness
        │
        │ ATOF
        ▼
consumer binary stdin

Relay could launch the consumer directly.

For more advanced composition:

parent process
    │
    ├── launches harness
    │       ├── fd 1/2: normal UI/output
    │       └── fd 3: ATOF
    │                │
    │                ▼
    └────────── consumer binary

the parent can create the pipe and pass its writable side to the harness as an inherited descriptor.

File descriptor numbers are process-local, so many harness instances can all use the same configured descriptor number, for example fd = 3, while each descriptor points to a different underlying pipe:

Harness A              Harness B              Harness C
   fd 3                   fd 3                   fd 3
    │                      │                      │
    ▼                      ▼                      ▼
  pipe A                 pipe B                 pipe C
    │                      │                      │
    ▼                      ▼                      ▼
consumer A             consumer B             consumer C

This makes inherited FDs particularly suitable for running many concurrent coding-agent harnesses without assigning unique ports, socket paths, FIFO names, or file paths.

These capabilities would make ATOF much easier to integrate with CLI tooling, coding-agent harnesses, evaluators, telemetry processors, test runners, local orchestration systems, and arbitrary user-defined binaries.

Proposed enhancement

Add built-in daemonless local ATOF sinks configurable through the existing observability TOML configuration.

The enhancement should include two first-class sink types:

1. A subprocess sink.
2. An inherited file-descriptor sink.

Optional stdout/stderr convenience configuration would also be useful.

1. Subprocess sink

Relay should be able to launch a configured executable and stream canonical ATOF events directly into the process's stdin.

For example:

[[components.config.atof.sinks]]
type = "process"
command = ["atof-consumer", "--some-option"]

Conceptually:

Relay-enabled harness
        │
        │ canonical ATOF NDJSON
        ▼
consumer process stdin

This should enable arbitrary local consumers such as:

• evaluation tools,
• telemetry processors,
• converters,
• database importers,
• CLI pipelines,
• local observability tools,
• or user-defined Rust/Go/C/Python/Node binaries.

The process should be managed as part of the sink lifecycle and should not require a daemon or network listener.

Potential optional configuration could include:

[[components.config.atof.sinks]]
type = "process"
command = ["atof-consumer", "--some-option"]
working_directory = "/path/to/project"
inherit_environment = true

Environment overrides could optionally be supported:

[components.config.atof.sinks.env]
FOO = "bar"

The exact TOML shape is not important to this proposal; the important capability is direct ATOF-to-child-stdin streaming.

2. Inherited file-descriptor sink

Relay should also support writing canonical ATOF events to an already-open file descriptor:

[[components.config.atof.sinks]]
type = "fd"
fd = 3

The descriptor would be supplied by the parent process or shell.

Relay should not need to know what the descriptor ultimately represents. It could point to:

• a pipe connected to another process,
• a file,
• a socket-backed descriptor,
• a FIFO,
• or another local IPC mechanism supported by the host OS.

For example:

parent
  │
  ├── create pipe
  │
  ├── launch harness
  │      └── map pipe write end → fd 3
  │
  └── launch consumer
         └── map pipe read end → stdin

Because descriptor numbers are local to each process, every harness instance can use the same configuration:

[[components.config.atof.sinks]]
type = "fd"
fd = 3

while receiving a different underlying pipe from its parent.

This avoids per-instance allocation of TCP ports, socket paths, FIFO names, or telemetry filenames.

3. Optional stdio convenience sinks

Convenience aliases could also be supported:

[[components.config.atof.sinks]]
type = "stdio"
target = "stdout"

or:

[[components.config.atof.sinks]]
type = "stdio"
target = "stderr"

These are convenience cases of local byte-stream output; the subprocess and inherited-FD sinks are the core requested primitives.

All local sinks should emit the canonical ATOF representation, ideally using the same NDJSON / JSONL framing semantics as existing ATOF output.

None of these modes should require a Relay daemon, TCP/HTTP/WebSocket listener, sidecar, collector, or temporary filesystem storage.

Runtime contract and binding impact

The core behavior should ideally live in the Rust runtime / observability layer so that Rust, Python, Node.js, and experimental bindings receive identical behavior through standard Relay configuration.

Users should not need to implement binding-specific subscribers to use these sinks.

Subprocess sink contract

For type = "process", the runtime should define:

• when the child process is started,
• whether startup is eager or lazy,
• how the executable and arguments are specified,
• how ATOF events are framed on stdin,
• buffering and flushing behavior,
• backpressure behavior when the consumer is slow,
• behavior if the consumer exits early,
• behavior on broken stdin / broken pipe,
• handling of child exit codes,
• whether the child is waited for during Relay shutdown,
• how the child is terminated if the harness exits,
• stdout/stderr handling for the child,
• environment inheritance and optional overrides,
• working-directory behavior,
• and whether automatic restart is supported.

A simple initial contract would be sufficient:

• start one child process per configured process sink,
• stream canonical ATOF NDJSON to its stdin,
• do not automatically restart it,
• detect and report premature process termination,
• and cleanly close stdin / reap the child during shutdown.

File-descriptor sink contract

For type = "fd", the runtime should define:

• descriptor ownership,
• behavior for invalid or closed descriptors,
• write failures,
• broken pipes,
• buffering and flushing,
• backpressure,
• shutdown behavior,
• and platform-specific behavior.

Relay should generally not close inherited descriptors it did not create unless ownership is explicitly transferred.

The same configured descriptor number must be usable across multiple concurrently running harness processes, since descriptor tables are process-local.

For example, multiple harnesses may all use:

type = "fd"
fd = 3

while their respective parent processes map a different pipe to fd 3 in each child.

Binding expectations

• Rust: implement the local sink behavior in the core runtime / observability implementation.
• Python: configuration should be sufficient; no Python callback should be required.
• Node.js: configuration should be sufficient; no Node-specific subscriber should be required.
• Experimental bindings / FFI: inherit the same runtime behavior where possible.

The ATOF representation and framing should remain consistent across bindings.

Alternatives considered

1. ATOF file sink

Works, but requires filesystem persistence and usually an additional follow/tail/read/delete lifecycle.

This is unnecessary when the desired destination is another local process.

It also introduces filename/path coordination when many harness instances are running concurrently.

2. HTTP / WebSocket / other network sinks

These work, but require a listener or collector and introduce networking for what may be purely local process composition.

Multiple concurrent harnesses may also require port allocation, endpoint discovery, or routing configuration.

3. Custom subscriber

A custom subscriber can manually spawn a process or write to an inherited descriptor, but every integration must implement and maintain code for behavior that is generic enough to be a built-in sink.

This also prevents the integration from being expressed entirely through plugins.toml.

4. Custom Relay plugin

A plugin could implement the same behavior, but requiring users to package and maintain a plugin merely to pipe canonical ATOF into another executable or existing descriptor adds unnecessary integration overhead.

5. Only support subprocess sinks

A subprocess sink is convenient when Relay should launch and own the consumer, but it does not cover applications where a parent process already owns the topology.

For example, a launcher may want to create pipes, supervise multiple harnesses and consumers, multiplex streams, or integrate with existing IPC infrastructure.

An inherited-FD sink covers this case cleanly.

6. Only support inherited file descriptors

An FD sink is flexible, but it requires the caller to construct the process topology.

For the very common case:

Relay -> consumer stdin

a first-class subprocess sink is substantially easier to configure and operate.

The two sinks are therefore complementary:

• process: Relay launches and owns the consumer.
• fd: the caller owns process topology and Relay writes to an existing descriptor.

7. Named pipes / Unix sockets as dedicated sink types

These could also solve some local IPC use cases, but an inherited file descriptor is a more general Unix primitive.

A parent process can create the desired pipe/socket and pass its descriptor to the harness without Relay needing separate sink implementations for each IPC mechanism.

Acceptance criteria

• ATOF can be streamed directly into another executable's stdin using standard NeMo Relay configuration.

• A subprocess sink can be configured through plugins.toml, with a command and argument list.

• The configured consumer process is started and managed as part of the sink lifecycle.

• Canonical ATOF events are written to the child process's stdin using a documented framing format, ideally NDJSON / JSONL.

• Child-process startup failure, premature exit, broken stdin, exit codes, buffering, backpressure, and shutdown behavior are documented and tested.

• ATOF can also be emitted to an already-open/inherited local file descriptor using standard Relay configuration.

• A parent process can create a pipe, pass the write-side descriptor to a Relay-enabled harness, and connect the read side to another binary.

• Multiple concurrent harness processes can use the same configured descriptor number, such as fd = 3, while each receives a different underlying descriptor from its parent.

• No globally unique FD numbers are required between harness processes.

• stdout/stderr are either directly supported as convenience sinks or usable through their conventional descriptors where supported.

• Descriptor ownership semantics are documented.

• Invalid descriptors, closed descriptors, broken pipes, write failures, buffering, flushing, backpressure, and shutdown behavior are documented and tested.

• No TCP/HTTP/WebSocket listener, daemon, sidecar, collector, temporary file, unique port, or unique socket path is required for these local sink modes.

• The functionality is available through supported Rust, Python, and Node.js runtimes without requiring user-written binding-specific subscriber code.

• Platform-specific behavior or limitations, including Windows behavior, are documented.

• Existing ATOF sinks remain backward compatible.

• At least one end-to-end example demonstrates:

Relay-enabled harness -> ATOF -> child process stdin

• At least one end-to-end example demonstrates:

parent process
    ├── harness fd 3
    │       │
    │       └── ATOF
    │
    └── consumer stdin

• At least one example demonstrates multiple concurrent harness instances all configured with the same FD number while using separate underlying pipes.

[willkill07]

A harness that embeds NeMo Relay can directly register its own subscribers.

Subscribers get every event that ATOF exporters do. May I ask what the material need/justification is?

[smallstepman]

Thanks - yes, I understand that harness already using NeMo Relay can register its own subscriber (which could e.g. implement tty-based sink).

The material need I'm trying to address is slightly different: being able to attach a local consumer without modifying the harness implementation or writing binding-specific subscriber code.

I want observability/export behavior to remain a deployment/configuration concern, just like the existing ATOF file/stream sinks are.

Concretely, I'd like to be able to take an arbitrary Relay-enabled harness and configure:

[[components.config.atof.sinks]]
type = "process"
command = ["my-consumer"]

or:

[[components.config.atof.sinks]]
type = "fd"
fd = 3

without adding Python/Node/Rust code to that harness.

The practical reasons are:

• I may not own the harness implementation. I may only control its Relay configuration.
• Separation of concerns. The harness emits Relay events; deployment decides where those events go. This is already the model for the existing ATOF exporters.
• CLI/Unix composition. A subprocess/stdin sink makes Relay -> arbitrary binary possible without introducing HTTP/WebSocket, a daemon, collector, or intermediate file.
• Standardized lifecycle semantics. If every consumer implements its own subscriber, each integration has to independently solve buffering, backpressure, broken pipes, shutdown/flush behavior, child-process lifecycle, etc. A built-in sink gives those semantics one implementation.

[smallstepman]

I didn't answer your main question... The main material need is:

I need daemonless way to tap into the event stream of an already Relay-enabled harness

I want to avoid requiring a collector, sidecar, network listener, or harness code changes just to consume the events locally. A process or inherited fd sink would make that possible purely through configuration.

[willkill07]

I need daemonless way to tap into the event stream of an already Relay-enabled harness

Fundamentally, this is why plugins exist. Dynamic plugin lifecycles are managed by the NeMo Relay runtime. We need/want consistency across all platforms. Your above proposal would prevent a configuration file from being portable across platforms.

without adding Python/Node/Rust code to that harness.

Do you have an example of such a harness that isn't implemented in one of those three languages? If a harness is Relay-enabled, it will have it.

CLI/Unix composition. A subprocess/stdin sink makes Relay -> arbitrary binary possible without introducing HTTP/WebSocket, a daemon, collector, or intermediate file.

I do not like the idea of Relay needing to own its own subprocess that could arbitrarily be anything. A malicious actor could insert a process invocation that does not read stdin at all.

Standardized lifecycle semantics. If every consumer implements its own subscriber, each integration has to independently solve buffering, backpressure, broken pipes, shutdown/flush behavior, child-process lifecycle, etc. A built-in sink gives those semantics one implementation.

I strongly disagree here. Relay will just send it regardless. there is no buffer. there is no flushing. You are putting buffering and backpressure on whatever downstream process.

[willkill07]

TL;DR - dynamic plugin registering its own subscriber fills your stated requirements without introducing platform-specific behavior. Plugins can be written in native Rust (native cdylib OR gRPC worker) or Python (gRPC worker). Plugins must be installed/managed by the user explicitly and have real attestation and optional signing requirements.

[smallstepman]

We need/want consistency across all platforms

fair, thanks for taking the time to consider this! I'll close the issue

Do you have an example of such a harness that isn't implemented in one of those three languages?

It's not really a language related issue. I'm building provenance cli tool that's closely tied to VCS, the requirement is that supported harness must have NeMo-Relay installed. Running a daemon just doesn't fit the planned architecture or expected usage of my tool. Hence I hoped for some harness 3> my-tool harness-hook so I could tap into events without any extra steps beyond adding additional sink in NeMo-Relay's config TOML.

[willkill07]

If it is Relay-specific, I do believe the plugin approach is best fit for your needs. A Rust native plugin has the advantage of running in the same process and offers the lowest overhead. If you wanted this plugin to manage separate process lifetime and execution, you can do that too.