[OPIK-7686] [DOCS] Document where the async-insert cutover bump lives

[thiagohora]

Details

Closes the follow-up @andrescrz raised on #7675: async_insert_busy_timeout_max_ms has to be increased during the traces cutover, and it was unsettled whether that belongs in the chart values or as a temporary env var on the deployed chart. Decision: the temporary env var — the option you leaned towards. No chart change, so the three ANALYTICS_DB_ASYNC_INSERT_* knobs stay out of values.yaml.

Docs-only, and scoped to the operator runbook rather than any published doc page: the bump is temporary, so it should not become official self-host documentation. One new section, "Where the buffer bump lives (and how to revert it)", plus cross-references from the four places that previously said "config, see below" without ever saying what to set.

The decision, and its rationale

• No chart change is needed — component.backend.env is a free-form map rendered straight into the backend ConfigMap.
• Removal is a clean one-step rollback. Unset means "leave queryParameters alone" (DatabaseAnalyticsFactory), so deleting the key restores the baked-in async_insert_busy_timeout_max_ms=250. There is no "set it back to 250" edit and no pinned value that can later drift from the config.yml default.
• The value is deployment- and window-specific, so it belongs with that deployment's own config — still version-controlled and auditable — rather than as a chart default every install inherits.
• A values knob would save no work — you edit the deployment config either way.

The section also corrects the record: the reason first given on #7675 for excluding these (that empty ConfigMap values "would not be inert") was wrong — config.yml already ships ${ANALYTICS_DB_ASYNC_INSERT_*:-} as the default, so the empty case is the normal path everywhere, and @Min(1) does not fire on a null Integer. Exposing them with empty defaults would have been safe. The exclusion now rests on reversibility and scope, not safety. Happy to add all three with empty defaults instead if you'd rather have them discoverable in the chart.

And the trap is written down explicitly: never bump it by editing ANALYTICS_DB_QUERY_PARAMETERS — that means re-pasting the whole tuning string and risks silently dropping one of the other settings.

Two operational facts the runbook did not state

1. It takes effect only on a backend restart, and that must be confirmed before continuing. The value arrives through the container environment (envFrom.configMapRef under Helm), injected at container start only — editing the ConfigMap does not reach a running pod. How the restart is triggered is deployment-specific and the chart ships no automation for it: some deployments run a ConfigMap watcher that rolls the workload, others need an explicit kubectl rollout restart. The runbook now says to know which case applies before the window, and either way to verify the ceiling is live on every instance rather than assume it — with the commands to do so. Getting this wrong is silent: you would enter the delta and EXCHANGE with the old 250ms ceiling still live.
2. The chosen ceiling must stay below the pod's termination grace period. The revert is delivered by a second restart, at a moment when pods hold inserts parked for up to the widened ceiling. The chart does not set terminationGracePeriodSeconds, so it is the Kubernetes default 30s — comfortably above ~10000ms, but a much larger ceiling would let SIGKILL land on parked inserts. Nothing previously connected those two numbers.

A planning consequence falls out of (1): traceColumnsNonNullable (prereq 7) is another entry in the same backend config and the same ConfigMap, and "The final cutover window" step 1 asks for both — so they should land together and ride a single restart. That note is now in both places.

Scope note

The section is deliberately deployment-agnostic — it names the key, the value, the revert and the constraints, not any particular environment's file paths. Comet's own deployment specifics are recorded internally instead. If these knobs later earn a spot in the published self-host docs, the self-hosted instructions can follow then; a temporary cutover-only bump does not warrant one now.

Change checklist

• User facing
• Documentation update

Issues

• Resolves #
• OPIK-7686

AI-WATERMARK

AI-WATERMARK: yes

• Tools: Claude Code
• Model(s): Claude Opus 5
• Scope: Investigated the config path and drafted the runbook section, the cross-references and the Go/No-Go item.
• Human verification: Author reviewed the diff and directed the scope — including the decision not to add the knobs to the chart, and the call to keep a temporary bump out of published documentation and out of environment-specific detail.

Testing

Docs-only — no code, no chart, no SQL touched, so nothing to run. The claims were verified rather than tested:

• envFrom.configMapRef: opik-backend confirmed as the backend's config path in the chart, and read back off a live deployment.
• terminationGracePeriodSeconds confirmed absent from the chart (hence the Kubernetes default 30s) and read back as 30 on a live deployment.
• Confirmed the chart ships no ConfigMap-watcher/reload automation and no podAnnotations default — which is why the restart trigger is documented as deployment-specific rather than automatic.
• A live backend ConfigMap holds none of the three ANALYTICS_DB_ASYNC_INSERT_* keys and no ANALYTICS_DB_QUERY_PARAMETERS, confirming deployments inherit the config.yml 250ms default today.
• The kubectl get pods -l component=opik-backend command quoted in the section was run as written and returns the expected rows.
• In-page anchor slug resolves; no cross-repo or environment-specific paths introduced.

Documentation

This PR is the documentation, and only the operator runbook: apps/opik-backend/data-migrations/traces-local-v2-cutover/README.md. No published doc page is touched, by design — see the scope note above.

[github-actions]

⏱️ pre-commit per-hook timing

No linted files changed — nothing to run.

⏭️ 43 skipped (no matching files changed)

Hook	Description	Result
🐍 trim trailing whitespace — python sdk	Strip trailing whitespace	⏭️
🐍 fix end of files — python sdk	Ensure files end in a newline	⏭️
🐍 ruff — python sdk	Lint + autofix Python (ruff)	⏭️
🐍 ruff-format — python sdk	Format Python code (ruff)	⏭️
🐍 mypy — python sdk	Static type check	⏭️
🤖 trim trailing whitespace — optimizer	Strip trailing whitespace	⏭️
🤖 fix end of files — optimizer	Ensure files end in a newline	⏭️
🤖 check yaml — optimizer	Validate YAML syntax	⏭️
🤖 check json — optimizer	Validate JSON syntax	⏭️
🤖 check toml — optimizer	Validate TOML syntax	⏭️
🤖 check for added large files — optimizer	Block large files (>1MB)	⏭️
🔐 detect private key — optimizer	Block committed private keys	⏭️
🤖 check for merge conflicts — optimizer	Block merge-conflict markers	⏭️
🤖 check for case conflicts — optimizer	Block case-only name clashes	⏭️
🤖 pyupgrade — optimizer	Modernize Python syntax	⏭️
🤖 ruff — optimizer	Lint + autofix Python (ruff)	⏭️
🤖 ruff-format — optimizer	Format Python code (ruff)	⏭️
🤖 mypy — optimizer	Static type check	⏭️
📓 nbstripout — optimizer notebooks	Strip notebook output	⏭️
📝 markdownlint — optimizer	Lint Markdown	⏭️
🔤 codespell — optimizer	Fix common misspellings	⏭️
📊 radon cc — optimizer	Cyclomatic-complexity gate	⏭️
📊 radon raw — optimizer	Raw size metrics gate	⏭️
📊 xenon — optimizer	Fail on complexity thresholds	⏭️
📊 lizard — optimizer	Cyclomatic-complexity gate	⏭️
🧹 vulture — optimizer	Find dead code	⏭️
🛡️ trim trailing whitespace — guardrails	Strip trailing whitespace	⏭️
🛡️ fix end of files — guardrails	Ensure files end in a newline	⏭️
🛡️ ruff — guardrails	Lint + autofix Python (ruff)	⏭️
🛡️ ruff-format — guardrails	Format Python code (ruff)	⏭️
🛡️ mypy — guardrails	Static type check	⏭️
⚓ helm-docs	Regenerate Helm chart README	⏭️
block non-public FE plugins	Block non-public FE plugins	⏭️
☕ spotless — java backend	Format Java code	⏭️
🧪 pre-commit wrapper smoke tests	Self-test the wrapper scripts	⏭️
🧪 rebaseline script tests	Self-test the changelog re-baseline script	⏭️
🌐 eslint — frontend	Lint + autofix JS/TS	⏭️
🌐 typecheck — frontend	Whole-project tsc type check	⏭️
📘 eslint — typescript sdk	Lint + autofix JS/TS	⏭️
📘 typecheck — typescript sdk	Whole-project tsc type check	⏭️
⚙️ actionlint — github workflows	Lint GitHub Actions workflows	⏭️
🐳 hadolint — dockerfiles	Lint Dockerfiles	⏭️
🌈 zizmor — github workflows security	Security-scan GitHub Actions workflows	⏭️

[CometActions]

No test needed here.

No product surface in this PR — only docs, CI, tests, lockfiles or other non-product files.

Run

Advisory, from the QA test radar. Nothing here blocks this PR, and anything it proposes is a draft for review.

Re-checked after a push on 20 Aug 11:58 UTC.