Put legacy layout hint under protected metadata

Author: bruceg

rfcs/2026-04-29-25329-trace-data-model.md

@@ -893,8 +893,8 @@ During the migration, `TraceEvent` is an enum:
 pub enum TraceEvent {
     /// Pre-migration source output: an untyped `LogEvent` whose key layout is
     /// defined by the producing source. The producing source identifies its
-    /// layout in `LogEvent.metadata().value()` under the reserved key
-    /// `_vector.trace_legacy_layout` so the correct shim can be selected
+    /// layout in `LogEvent.metadata().value()` under the reserved sub-key
+    /// `vector.trace_legacy_layout` so the correct shim can be selected
     /// after fan-in, disk-buffer round-trips, or `vector` source/sink hops.
     Legacy(LogEvent),
     /// Post-migration typed container.
@@ -908,32 +908,11 @@ pub enum TraceEvent {
 }
 ```
 
-The shim dispatch needs a stable producer identifier that survives every transport step a
-`Legacy` event takes inside Vector: fan-in, disk buffers, and `vector` source/sink hops.
-`EventMetadata.source_type` is not viable on its own: the topology source pump in
-`src/topology/builder.rs` (the `run_source_output_pump` task) unconditionally calls
-`metadata.set_source_type(source_type)` on every event a source emits, so a `Legacy` trace
-deserialized by the `vector` source has its upstream `source_type` (`"opentelemetry"` or
-`"datadog_agent"`) rewritten to `"vector"` and the original layout identifier is lost. The
-overwrite is correct for the field's documented role (it identifies the source that produced
-the local emission), but it forecloses using `source_type` as the shim selector across a
-serialised hop.
-
-The migration carries the layout hint in the existing
-`EventMetadata.value` arbitrary-metadata `Value` instead, under the reserved key
-`_vector.trace_legacy_layout`. The `opentelemetry` and `datadog_agent` sources each set this
-key to a static string identifying themselves on every `Legacy` trace they emit; the source
-pump does not touch `EventMetadata.value`, and the metadata `Value` is serialised end-to-end
-by the existing `Metadata`/`metadata_full` proto encoding, so the hint survives any number of
-intermediate hops including `vector` source/sink serialisation. The shim dispatch reads this
-key to select the correct `Legacy -> Typed` conversion. A `Legacy` event whose hint is absent
-or maps to no registered shim returns an error and emits a rate-limited `warn!` log rather
-than guessing the layout.
-
-The reserved key disappears with the `Legacy` variant: once every trace source has migrated to
-native `Typed` output, the sources stop setting it, the shim dispatch is removed, and no
-production traffic carries it. No permanent type-system or wire-format additions are required;
-the convention lives only as long as the migration enum does.
+Trace event-producing sources each set the reserved sub-key `vector.trace_legacy_layout` in
+`EventMetadata.value` to a static string identifying themselves on every `Legacy` trace they emit.
+`to_typed()` reads this sub-key to select the corresponding `Legacy -> Typed` shim. A `Legacy` event
+whose hint is absent or maps to no registered shim returns an error and emits a rate-limited `warn!`
+log.
 
 The end-state `struct TraceEvent { resource, scope, chunk, spans, metadata }` shown above is
 reached by deleting the `Legacy` arm once every component has migrated; the `Typed` arm's
@@ -957,7 +936,7 @@ Both accessor families coexist on `TraceEvent` and dispatch on the variant:
   zero-overhead post-migration signature. The panic is the loud failure that the intake-convert
   pattern below avoids.
 - Explicit `to_typed(&mut self)` rewrites a `Legacy` variant in place into `Typed` by reading the
-  `_vector.trace_legacy_layout` hint from `EventMetadata.value` and invoking the corresponding
+  `vector.trace_legacy_layout` hint from `EventMetadata.value` and invoking the corresponding
   source-specific shim. Already-`Typed` events are a no-op. There is no symmetric `to_legacy`.
 
 Per-component shims are unidirectional (`Legacy -> Typed` only). The `datadog_agent` source
@@ -1125,16 +1104,18 @@ are deleted, leaving only the typed struct.
   etc.) are removed from `TraceEvent` before the source steps; every remaining call site then
   fails to compile, making the consumer migration a mechanical fix-the-build task rather than a
   runtime-failure audit.
-- Shim selection is keyed on a reserved key `_vector.trace_legacy_layout` carried in
-  `EventMetadata.value` and set by the producing trace source. The metadata `Value` is serialized
-  with every event record and passes through fan-in, disk buffers, and `vector` source/sink hops
-  unchanged (unlike `EventMetadata.source_type`, which the topology source pump rewrites on every
-  emission and so cannot serve as the selector across a serialised hop). Conversion is invoked
-  explicitly by `to_typed(&mut self)`; immutable typed accessors panic on `Legacy` rather than
-  converting on demand, because returning typed references through a `&self` accessor would
-  require either mutating `self` or returning owned/`Cow` shapes that would have to be torn out
-  again post-migration. The convention lives only for the duration of the migration and
-  disappears with the `Legacy` variant; no new struct field or wire-format extension is needed.
+- Shim selection is keyed on a reserved sub-key `vector.trace_legacy_layout` in
+  `EventMetadata.value` set by the producing trace source. The `vector` metadata namespace is
+  read-only to VRL programs (configured by `compile_vrl`), so transforms between source and sink
+  cannot accidentally delete or overwrite the hint. The metadata `Value` is serialized with every
+  event record and passes through fan-in, disk buffers, and `vector` source/sink hops unchanged
+  (unlike `EventMetadata.source_type`, which the topology source pump rewrites on every emission
+  and so cannot serve as the selector across a serialised hop). Conversion is invoked explicitly
+  by `to_typed(&mut self)`; immutable typed accessors panic on `Legacy` rather than converting on
+  demand, because returning typed references through a `&self` accessor would require either
+  mutating `self` or returning owned/`Cow` shapes that would have to be torn out again
+  post-migration. The convention lives only for the duration of the migration and disappears with
+  the `Legacy` variant; no new struct field or wire-format extension is needed.
 
 ## Drawbacks
 
@@ -1442,14 +1423,14 @@ shims are unidirectional (`Legacy -> Typed` only) and a `Typed` event has no sou
 on which to base a `Typed -> Legacy` conversion.
 
 - [ ] Land the legacy-layout hint as a precursor: the `opentelemetry` and `datadog_agent` sources
-  each set `EventMetadata.value`'s reserved key `_vector.trace_legacy_layout` to a static string
+  each set `EventMetadata.value`'s reserved sub-key `vector.trace_legacy_layout` to a static string
   identifying themselves on every trace event they emit. Purely additive -- no consumer reads the
   key yet. See "Migration: coexistence of `LogEvent` and typed representations" for the rationale.
 - [ ] Convert `TraceEvent` to the migration enum: `Legacy(LogEvent)` and `Typed { resource, scope,
   chunk, spans, metadata }`, demonstrating the new structure and the supporting types. Add the
   explicit `to_typed(&mut self)` converter that reads
-  `EventMetadata.value."_vector.trace_legacy_layout"` and invokes the corresponding `Legacy ->
-  Typed` shim. Typed accessors (`resource()`, `spans()`, ...) panic on `Legacy` with a diagnostic
+  `EventMetadata.value` sub-key `vector.trace_legacy_layout` and invokes the corresponding
+  `Legacy -> Typed` shim. Typed accessors (`resource()`, `spans()`, ...) panic on `Legacy` with a diagnostic
   pointing at `to_typed()`. Every component still produces and consumes `Legacy`; nothing
   functionally changes. A `to_typed()` call on an event whose layout hint is absent or unrecognized
   returns an error and emits a rate-limited `warn!` log.
@@ -1466,7 +1447,7 @@ on which to base a `Typed -> Legacy` conversion.
     `component_errors_total{error_type="unsupported_payload_version"}` increment and a
     rate-limited error log instead of being silently translated.
 - [ ] OTLP `Legacy -> Typed` shim and `Typed -> OTLP-wire` conversion in `lib/opentelemetry-proto`.
-  The shim dispatch infrastructure uses the `_vector.trace_legacy_layout` hint in
+  The shim dispatch infrastructure uses the `vector.trace_legacy_layout` hint in
   `EventMetadata.value` to select the OTLP shim on demand. The `Typed -> wire` conversion is what
   the eventual native source emission and any OTLP sink will use.
 - [ ] Remove the legacy `tracerPayloads`-empty ingest branch from