Skip to content

Commit 380ef57

Browse files
Minipadaclaude
andcommitted
feat(demos,deploy,docs): migrate off blessed postgres/s3/console (#471)
Per ADR-0003 and the #470 recipes, postgres/s3(records)/console are pure Vector-sink wrappers with no DC-specific logic, so every demo, deploy param file and doc example that configured them via the blessed ROS-param form now uses the ADR-0003 passthrough (custom_config_files) instead, ahead of #472's removal of that blessed code path. What moved: dc_demos/params/*.yaml (10 demos), deploy/robot/params/ aio_params_local.yaml, and the doc pages listing worked examples of these Destination types (uptime_stdout, elasticsearch, mcap_recording, fastdds_stats_pgsql_grafana, qrcodes_minio_pgsql, tb3_aws_minio_pgsql, tb3_aws_influxdb's stale cross-reference, configuration_examples, concepts, raw_topics, demos). A `file` Destination is the passthrough's required anchor throughout (dc_bridge derives its ROS subscriptions and dc.<tag> routes from `destinations` alone, never from a snippet's `inputs`). Two things deliberately did not move, both verified against dc_bridge's own source, not assumed: - `s3`/`receives: files` stays blessed everywhere it appears (qrcodes_minio_ pgsql, tb3_simulation_pgsql_minio, camera.md, configuration_examples Example 9): it is served entirely by dc_uploader reading these same ROS params, never by a Vector sink, so there is no passthrough equivalent. - deploy/robot/params/robot_params_local.yaml keeps blessed `postgres`: that scenario runs `shipper.managed: false` (the three-container split), and custom_config_files snippets are only wired into the config reaching the separate `vector` container in *managed* mode -- confirmed by actually standing the split topology up with a passthrough sink and finding no `pgsql` sink in the rendered vector.toml at all. Left blessed with a comment explaining the gap rather than shipping a silently-broken config; worth its own follow-up issue against dc_bridge/the split-deployment compose+quadlet units. Verified for real, not just visually inspected: every touched dc_demos demo was launched (Gazebo-based ones DC-only, synthetic ROS messages standing in for Nav2/camera output per the issue's own guidance, since navigation behavior is untouched by this change) and its passthrough sink's actual delivery confirmed -- rows in PostgreSQL, documents in a live Elasticsearch, the .mcap file and console/file output. aio_params_local.yaml's managed-mode passthrough was validated end to end via its real compose stack, including `vector validate` against the rendered config plus the passthrough snippet. Closes #471 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WYVDzgJx2xLAzixNFm7b7H Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
1 parent 654a348 commit 380ef57

33 files changed

Lines changed: 749 additions & 259 deletions
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `postgres` Destination, for the
5+
# fastdds_stats_pgsql_grafana demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/fastdds_stats_pgsql_grafana_sink.toml" ~/.dc/
10+
11+
[sinks.pgsql]
12+
type = "postgres"
13+
inputs = ["dc.dc.measurement.fastdds_stats"]
14+
endpoint = "postgres://dc:password@127.0.0.1:5432/dc" # user:password@host:port/database
15+
table = "dc"
16+
17+
[sinks.pgsql.buffer]
18+
type = "disk"
19+
max_size = 268435488 # Vector's disk-buffer minimum; a passthrough sink gets none by default
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `console` Destination, for the
5+
# group_memory_uptime_stdout demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/group_memory_uptime_stdout_sink.toml" ~/.dc/
10+
11+
[sinks.debug_console]
12+
type = "console"
13+
inputs = ["dc.dc.group.memory_uptime"]
14+
target = "stdout"
15+
16+
[sinks.debug_console.encoding]
17+
codec = "json"
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `postgres` Destinations, for the
5+
# qrcodes_minio_pgsql demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/qrcodes_minio_pgsql_sink.toml" ~/.dc/
10+
11+
[sinks.pgsql]
12+
type = "postgres"
13+
inputs = [
14+
"dc.dc.measurement.map",
15+
"dc.dc.measurement.right_camera",
16+
"dc.dc.measurement.left_camera",
17+
"dc.dc.group.robot",
18+
]
19+
endpoint = "postgres://dc:password@127.0.0.1:5432/dc" # user:password@host:port/database
20+
table = "dc"
21+
22+
[sinks.pgsql.buffer]
23+
type = "disk"
24+
max_size = 268435488 # Vector's disk-buffer minimum; a passthrough sink gets none by default
25+
26+
# File status Records (ADR-0005): `records_log`'s params/qrcodes_minio_pgsql.yaml block
27+
# names it as `files.metadata_destination`, which appends the `dc.files` Tag to its
28+
# routed set -- consumed here exactly like any other `dc.<tag>` route.
29+
[sinks.pgsql_files]
30+
type = "postgres"
31+
inputs = ["dc.dc.files"]
32+
endpoint = "postgres://dc:password@127.0.0.1:5432/dc"
33+
table = "dc_files"
34+
35+
[sinks.pgsql_files.buffer]
36+
type = "disk"
37+
max_size = 268435488
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `console` Destination, for the
5+
# qrcodes_stdout demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/qrcodes_stdout_sink.toml" ~/.dc/
10+
11+
[sinks.debug_console]
12+
type = "console"
13+
inputs = ["dc.dc.measurement.right_camera", "dc.dc.measurement.left_camera"]
14+
target = "stdout"
15+
16+
[sinks.debug_console.encoding]
17+
codec = "json"
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `postgres` Destinations, for the
5+
# tb3_simulation_pgsql_minio demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/tb3_simulation_pgsql_minio_sink.toml" ~/.dc/
10+
11+
[sinks.pgsql]
12+
type = "postgres"
13+
inputs = [
14+
"dc.dc.measurement.cpu",
15+
"dc.dc.measurement.memory",
16+
"dc.dc.measurement.os",
17+
"dc.dc.measurement.uptime",
18+
"dc.dc.measurement.camera",
19+
"dc.dc.measurement.cmd_vel",
20+
"dc.dc.measurement.distance_traveled",
21+
"dc.dc.measurement.driving_type",
22+
"dc.dc.measurement.position",
23+
"dc.dc.measurement.speed",
24+
"dc.dc.measurement.map",
25+
"dc.dc.measurement.rustfs_health",
26+
"dc.dc.measurement.pgsql_health",
27+
]
28+
endpoint = "postgres://dc:password@127.0.0.1:5432/dc" # user:password@host:port/database
29+
table = "dc"
30+
31+
[sinks.pgsql.buffer]
32+
type = "disk"
33+
max_size = 268435488 # Vector's disk-buffer minimum; a passthrough sink gets none by default
34+
35+
# File status Records (ADR-0005): `records_log`'s params/tb3_simulation_pgsql_minio.yaml
36+
# block names it as `files.metadata_destination`, which appends the `dc.files` Tag to
37+
# its routed set -- consumed here exactly like any other `dc.<tag>` route.
38+
[sinks.pgsql_files]
39+
type = "postgres"
40+
inputs = ["dc.dc.files"]
41+
endpoint = "postgres://dc:password@127.0.0.1:5432/dc"
42+
table = "dc_files"
43+
44+
[sinks.pgsql_files.buffer]
45+
type = "disk"
46+
max_size = 268435488
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `console` Destination, for the
5+
# tb3_simulation_stdout demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/tb3_simulation_stdout_sink.toml" ~/.dc/
10+
11+
[sinks.debug_console]
12+
type = "console"
13+
inputs = ["dc.dc.group.robot", "dc.dc.measurement.map"]
14+
target = "stdout"
15+
16+
[sinks.debug_console.encoding]
17+
codec = "json"
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `console` Destination, for the
5+
# uptime_custom_stdout demo -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/uptime_custom_stdout_sink.toml" ~/.dc/
10+
11+
[sinks.debug_console]
12+
type = "console"
13+
inputs = ["dc.dc.measurement.uptime_custom"]
14+
target = "stdout"
15+
16+
[sinks.debug_console.encoding]
17+
codec = "json"
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
2+
# SPDX-License-Identifier: MPL-2.0
3+
4+
# ADR-0003 passthrough equivalent of the blessed `console` Destination, for the
5+
# uptime_stdout demo (doc/src/dc/demos/uptime_stdout.md) -- see
6+
# doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough.
7+
#
8+
# Copied into place before launching:
9+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/uptime_stdout_sink.toml" ~/.dc/
10+
11+
[sinks.debug_console]
12+
type = "console"
13+
inputs = ["dc.dc.measurement.uptime"]
14+
target = "stdout"
15+
16+
[sinks.debug_console.encoding]
17+
codec = "json"

dc_demos/params/elasticsearch.yaml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,14 @@ dc_bridge:
1414
# `destinations` still has to name at least one blessed Destination, and the routes
1515
# the snippet consumes exist only for topics listed in *its* `inputs` — dc_bridge
1616
# derives both its ROS subscriptions and its route branches from `destinations`, and
17-
# never reads the snippet's `inputs`. `console` is the cheapest thing to put there,
18-
# and doubles as a local view of the same Records that land in Elasticsearch.
19-
destinations: ["console"]
20-
console:
21-
type: console
17+
# never reads the snippet's `inputs`. `console` used to be the cheapest thing to put
18+
# here, doubling as a local view of the same Records that land in Elasticsearch;
19+
# blessed `console` is retired in favor of the ADR-0003 passthrough recipe (#471,
20+
# see doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough), so
21+
# `file` is the anchor now instead.
22+
destinations: ["records_log"]
23+
records_log:
24+
type: file
2225
receives: records
2326
inputs:
2427
[
@@ -27,6 +30,7 @@ dc_bridge:
2730
"/dc/measurement/os",
2831
"/dc/measurement/uptime",
2932
]
33+
path: "/tmp/dc/elasticsearch_records.ndjson"
3034
time_key: "date"
3135
time_format: "double"
3236
custom_config_files: ["$HOME/.dc/elasticsearch_sink.toml"]

dc_demos/params/fastdds_stats_pgsql_grafana.yaml

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,21 @@ dc_bridge:
55
ros__parameters:
66
shipper:
77
data_dir: "$HOME/.dc/buffer"
8-
destinations: ["pgsql"]
9-
pgsql:
10-
type: postgres
8+
# blessed `postgres` is retired in favor of the ADR-0003 passthrough recipe (#471,
9+
# see doc/src/dc/destinations.md#recipes-postgres-s3-console-via-passthrough); `file`
10+
# is the anchor Destination `destinations` still needs -- dc_bridge derives its ROS
11+
# subscriptions and `dc.<tag>` routes from `destinations` alone, never from a
12+
# passthrough snippet's `inputs`. Copy the passthrough sink into place first:
13+
# mkdir -p ~/.dc && cp "$(ros2 pkg prefix dc_demos)/share/dc_demos/config/fastdds_stats_pgsql_grafana_sink.toml" ~/.dc/
14+
destinations: ["records_log"]
15+
records_log:
16+
type: file
1117
receives: records
1218
inputs: ["/dc/measurement/fastdds_stats"]
13-
host: "127.0.0.1"
14-
port: 5432
15-
user: "dc"
16-
password: "password"
17-
database: "dc"
18-
table: "dc"
19+
path: "/tmp/dc/fastdds_stats_pgsql_grafana_records.ndjson"
1920
time_key: "date"
2021
time_format: "double"
22+
custom_config_files: ["$HOME/.dc/fastdds_stats_pgsql_grafana_sink.toml"]
2123
vector_forward_host: "127.0.0.1"
2224
vector_forward_port: 24224
2325

0 commit comments

Comments
 (0)