Skip to content

Commit 654a348

Browse files
Minipadaclaude
andcommitted
docs(destinations): add passthrough recipes for postgres/s3/console
Per ADR-0003, postgres/s3/console are pure Vector-sink wrappers with no DC-specific logic, so anyone using the blessed form today can migrate to raw Vector `custom_config_files` passthrough now, ahead of #471/#472's code removal. Each recipe reproduces exactly what dc_bridge itself renders for the blessed form and was verified end-to-end against a live Vector 0.57.0 binary, real PostgreSQL and RustFS containers, and a synthetic fluent-forward event (ingested row/object/console line confirmed, not just written by inspection). Also documents a real gap found while verifying: Vector 0.57.0 gates its own ${VAR} config interpolation behind VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION, which dc_bridge never sets for the vendored Vector process it execs, so a passthrough snippet's ${VAR} silently stays literal unless that env var is set on the Bridge process itself. Adds doc/src/dc/migration.md as the landing page for this and future blessed-to-passthrough moves, linked from destinations.md and SUMMARY.md. Closes #470 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LXd466a9kszZJxUR51ey7 Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
1 parent 659d530 commit 654a348

3 files changed

Lines changed: 125 additions & 0 deletions

File tree

doc/src/SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@
7777
- [Destinations](./dc/destinations.md)
7878
- [KPI views](./dc/kpi_views.md)
7979
- [Configuration examples](./dc/configuration_examples.md)
80+
- [Migration](./dc/migration.md)
8081
- [Infrastructure setup](./dc/infrastructure_setup.md)
8182
- [Adminer](./dc/infrastructure_setup/adminer.md)
8283
- [Elasticsearch](./dc/infrastructure_setup/elasticsearch.md)

doc/src/dc/destinations.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,101 @@ against a simulated robot. [MCAP recording](./demos/mcap_recording.md)
298298
passthrough consumed by a standalone process instead of a Vector-native sink — the shape
299299
to follow for any store Vector has no sink for at all.
300300

301+
### Recipes: `postgres`, `s3`, `console` via passthrough
302+
303+
Per ADR-0003, `postgres`, `s3` and `console` are pure Vector-sink wrappers with no
304+
DC-specific logic layered on top — Vector's own `vector validate` already gives clear,
305+
field-level errors for these three sinks, so passthrough loses nothing on the validation
306+
front for this subset. Anyone using the blessed form today can move to passthrough now,
307+
before the blessed code path for these three types is removed
308+
([#471](https://github.com/minipada/ros2_data_collection/issues/471),
309+
[#472](https://github.com/minipada/ros2_data_collection/issues/472)). Each recipe below
310+
reproduces exactly what `dc_bridge` itself renders for the equivalent blessed
311+
configuration in [Configuration contract](#configuration-contract) above — confirmed by
312+
running Vector 0.57.0 against a live PostgreSQL and RustFS instance, not just written by
313+
inspection.
314+
315+
As with any passthrough, at least one blessed Destination is still needed to create the
316+
`dc.<tag>` route the snippet consumes — `console` is the cheapest (see
317+
[above](#passthrough-custom_config_files)). If you're migrating the `console` Destination
318+
itself, keep a `file` Destination (or another cheap blessed type) as the route anchor
319+
instead.
320+
321+
**`postgres`** — the blessed form's `host`/`port`/`user`/`password`/`database` collapse
322+
into a single connection-string `endpoint`; `table` is unchanged:
323+
324+
```toml
325+
# ~/.dc/postgres_sink.toml — passthrough equivalent of the blessed `pgsql` Destination
326+
[sinks.pgsql]
327+
type = "postgres"
328+
inputs = ["dc.dc.measurement.uptime"] # the public dc.<tag> route
329+
endpoint = "postgres://dc:${DC_PG_PASSWORD}@127.0.0.1:5432/dc" # user:password@host:port/database
330+
table = "dc"
331+
332+
[sinks.pgsql.buffer]
333+
type = "disk"
334+
max_size = 268435488 # Vector's disk-buffer minimum; a passthrough sink gets none by default
335+
```
336+
337+
If `user` or `password` contain characters reserved in a URI (`:`, `@`, `/`, `%`),
338+
percent-encode them yourself — `dc_bridge` does this automatically when rendering the
339+
blessed form, but a passthrough `endpoint` is handed to Vector verbatim.
340+
341+
**`s3`** — Vector's own sink id is `aws_s3`, not `s3`; credentials move under
342+
`[sinks.<name>.auth]` and `batch_timeout_secs` becomes `[sinks.<name>.batch] timeout_secs`:
343+
344+
```toml
345+
# ~/.dc/s3_sink.toml — passthrough equivalent of the blessed `rustfs` Destination
346+
[sinks.rustfs]
347+
type = "aws_s3" # Vector's sink id — not "s3"
348+
inputs = ["dc.dc.measurement.uptime"]
349+
bucket = "dc-records"
350+
endpoint = "http://127.0.0.1:9000" # omit for AWS S3
351+
region = "us-east-1"
352+
key_prefix = "robot1/"
353+
force_path_style = true # path-style addressing for self-hosted stores
354+
355+
[sinks.rustfs.auth]
356+
access_key_id = "rustfsadmin"
357+
secret_access_key = "${DC_S3_SECRET}"
358+
359+
[sinks.rustfs.batch]
360+
timeout_secs = 60 # object write interval; Vector default 300
361+
362+
[sinks.rustfs.buffer]
363+
type = "disk"
364+
max_size = 268435488
365+
366+
[sinks.rustfs.encoding]
367+
codec = "json"
368+
```
369+
370+
**`console`** — has no required fields either way, so this recipe mostly matters for
371+
consistency with the other two once the blessed path is gone:
372+
373+
```toml
374+
# ~/.dc/console_sink.toml — passthrough equivalent of the blessed `console` Destination
375+
[sinks.debug_console]
376+
type = "console"
377+
inputs = ["dc.dc.measurement.uptime"]
378+
target = "stdout" # or "stderr"
379+
380+
[sinks.debug_console.encoding]
381+
codec = "json"
382+
```
383+
384+
```admonish warning
385+
Unlike the blessed form's `password`/`secret_access_key` (expanded by `dc_bridge` itself
386+
before it ever writes a config file), the `${VAR}` references above are Vector's *own*
387+
interpolation and are otherwise off in the vendored Vector 0.57.0 binary `dc_bridge`
388+
spawns — a snippet's `${VAR}` is left as a literal string, silently sent as the password
389+
verbatim (or rejected outright, if it contains a reserved URI character like the braces
390+
here). Set `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` in the environment that
391+
launches the Bridge process itself (the vendored Vector inherits it) to make Vector honor
392+
`${VAR}` inside passthrough snippet content. Without it, put the literal secret in the
393+
file and rely on filesystem permissions instead.
394+
```
395+
301396
## File uploads: `receives: files` (the Uploader, [ADR-0005](./adr/0005-file-uploads-are-bridge-responsibility.md), [ADR-0014](./adr/0014-uploader-runs-as-its-own-process.md))
302397

303398
A Destination with `receives: files` (only `type: s3` qualifies) is served by the

doc/src/dc/migration.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Migration
2+
3+
This page tracks in-progress moves between configuration shapes DC itself is making —
4+
distinct from [Configuration examples](./configuration_examples.md), which teaches the
5+
current shape from scratch.
6+
7+
## Blessed `postgres`/`s3`/`console` &rarr; passthrough
8+
9+
[ADR-0003](./adr/0003-blessed-destinations-plus-passthrough.md) blesses `postgres`, `s3`,
10+
`file`, `console` and `vector` with a ROS-param form rendered into Vector config by
11+
`dc_bridge`. An audit of that blessed set found `postgres`, `s3` and `console` carry no
12+
DC-specific logic — they are pure Vector-sink wrappers, and Vector's own `vector validate`
13+
already gives clear, field-level errors for them. `dc_bridge`'s blessed code path for
14+
these three types is being removed
15+
([#472](https://github.com/minipada/ros2_data_collection/issues/472)), once every
16+
in-repo demo, deploy param file and doc using them is moved to passthrough
17+
([#471](https://github.com/minipada/ros2_data_collection/issues/471)). `file` and
18+
`vector` are unaffected — `file` drives the Uploader
19+
([ADR-0005](./adr/0005-file-uploads-are-bridge-responsibility.md)) and `vector` is
20+
reserved for the split-deployment/fleet work in
21+
[#440](https://github.com/minipada/ros2_data_collection/issues/440); neither is a plain
22+
sink wrapper.
23+
24+
**You do not need to wait for #471/#472.** Working passthrough recipes for all three
25+
types — reproducing exactly what `dc_bridge` renders for the blessed form today, verified
26+
against a live Vector instance — are in
27+
[Destinations: Recipes](./destinations.md#recipes-postgres-s3-console-via-passthrough).
28+
Point `custom_config_files` at one of them and drop the equivalent blessed Destination
29+
block; nothing else about your Measurements or routing changes.

0 commit comments

Comments
 (0)