Problem
deploy.sh in catalyst-otel-private validates the merged collector config and reports success, but it does not apply a collector-config change. A config-only deploy therefore ships inert — and reports green while doing so.
The final step is docker compose --project-directory "$PUBLIC_DIR" up -d. Compose only recreates a container when its service definition or image changes. collector-config.yaml and private/collector-config-private.yaml are bind-mounted, so editing them changes nothing compose can see, and the running collector keeps its old in-memory config.
Evidence (measured 2026-08-24, CTC-969 reconciliation)
After merging catalyst-otel-private#12 (which adds the metrics/cloudflare_otlp pipeline) and reconciling the box to main, deploy.sh ran to completion:
==> validating merged collector config (otelcol validate)
+ merged collector config is valid
==> bringing the merged stack up
Container otel-collector Running <-- NOT recreated
+ stack up
otel-collector still showed Up 38 hours, and the change was verifiably not applied:
POST https://otel-collector.catalystcloud.dev/v1/metrics -> HTTP 404
An explicit docker compose restart otel-collector fixed it immediately:
POST .../v1/metrics -> HTTP 200 (no-auth control still 403)
Why this matters
This is the same shape as the two prior no-op-gate bugs recorded in deploy.sh's own comments: a step that looks like it enforces something, reports success, and enforces nothing. Here the validate gate is real but the apply is missing, so the failure is invisible — dashboards and probes look unchanged, which reads as "the emitter isn't running" rather than "the config never loaded".
.agents/skills/otel-stack-ops/SKILL.md already documents the correct rule in its change→restart matrix (Collector pipeline → restart: yes). deploy.sh does not implement it.
Acceptance criteria
Given a change to collector-config.yaml or private/collector-config-private.yaml
When deploy.sh completes successfully
Then the running collector is serving the new config — not the config it had before
Given a deploy where no collector config changed
When deploy.sh runs
Then it does not needlessly restart the collector (avoid dropping in-memory metric state on unrelated deploys)
Given the collector is restarted by the deploy
When the script finishes
Then it reports the restart explicitly, so an operator can tell an applied change from a no-op run
Notes
Likely shape: hash the merged config (or the two source files) before and after, and docker compose restart otel-collector when it changed — or unconditionally up -d --force-recreate otel-collector. Prefer the targeted service over the whole stack: the deploy host runs ~20 unrelated containers (homeassistant, esphome, fluentd, …).
A restart clears the prometheus exporter's in-memory state, so claude_code_* series drop to zero and rebuild over ~10-15m (metric_expiration: 15m). That is expected and should be noted in the script's output so nobody reads it as an ingestion outage. Loki is unaffected.
Problem
deploy.shincatalyst-otel-privatevalidates the merged collector config and reports success, but it does not apply a collector-config change. A config-only deploy therefore ships inert — and reports green while doing so.The final step is
docker compose --project-directory "$PUBLIC_DIR" up -d. Compose only recreates a container when its service definition or image changes.collector-config.yamlandprivate/collector-config-private.yamlare bind-mounted, so editing them changes nothing compose can see, and the running collector keeps its old in-memory config.Evidence (measured 2026-08-24, CTC-969 reconciliation)
After merging catalyst-otel-private#12 (which adds the
metrics/cloudflare_otlppipeline) and reconciling the box to main,deploy.shran to completion:otel-collectorstill showedUp 38 hours, and the change was verifiably not applied:An explicit
docker compose restart otel-collectorfixed it immediately:Why this matters
This is the same shape as the two prior no-op-gate bugs recorded in
deploy.sh's own comments: a step that looks like it enforces something, reports success, and enforces nothing. Here the validate gate is real but the apply is missing, so the failure is invisible — dashboards and probes look unchanged, which reads as "the emitter isn't running" rather than "the config never loaded"..agents/skills/otel-stack-ops/SKILL.mdalready documents the correct rule in its change→restart matrix (Collector pipeline → restart: yes).deploy.shdoes not implement it.Acceptance criteria
Given a change to
collector-config.yamlorprivate/collector-config-private.yamlWhen
deploy.shcompletes successfullyThen the running collector is serving the new config — not the config it had before
Given a deploy where no collector config changed
When
deploy.shrunsThen it does not needlessly restart the collector (avoid dropping in-memory metric state on unrelated deploys)
Given the collector is restarted by the deploy
When the script finishes
Then it reports the restart explicitly, so an operator can tell an applied change from a no-op run
Notes
Likely shape: hash the merged config (or the two source files) before and after, and
docker compose restart otel-collectorwhen it changed — or unconditionallyup -d --force-recreate otel-collector. Prefer the targeted service over the whole stack: the deploy host runs ~20 unrelated containers (homeassistant, esphome, fluentd, …).A restart clears the prometheus exporter's in-memory state, so
claude_code_*series drop to zero and rebuild over ~10-15m (metric_expiration: 15m). That is expected and should be noted in the script's output so nobody reads it as an ingestion outage. Loki is unaffected.