Replace the single ClickHouse user with a least-privilege user model

[toffer]

What this does

Replaces the single, broadly-privileged otel ClickHouse user with a least-privilege model — one user per access path — enforced declaratively via ClickHouse config-level grants in the ClickHouseInstallation. The stock default superuser stays removed, and each component authenticates as a user scoped to exactly what it does.

Users

User	Role
schema_owner	Runs the schema-migration Job (DDL, telemetry/system-log TTLs, SYSTEM FLUSH LOGS) and is the materialized-view DEFINER.
otel	The OTel collector's ingest identity. Can be tightened to INSERT-only on the source tables via clickhouse.otel.restrictGrants.
llm_worker	The llm-worker Deployment — reads and appends the LLM job queue only.
monte_carlo	Monte Carlo's read-only data-source identity and the agent's queue producer: reads everything, writes only the queue + eval scores, never telemetry.
readonly_user	Optional SELECT-only user (readonly=2) for human / MCP / JDBC clients (e.g. DataGrip).
admin	Optional, gated break-glass superuser — reachable only over loopback (pod-exec) by default.

A shared "reader bundle" — SELECT on the telemetry DB plus the system / information_schema metadata reads JDBC/MCP clients need — is defined once and shared by monte_carlo and readonly_user.

How it's enforced and verified

• Privileges come from config-level <grants> in the CHI (declarative, applied by the operator) — no SQL-RBAC bootstrap user.
• The materialized views are pinned to DEFINER = schema_owner (new sql/0014 migration reparents existing installs in place), so the ingest user needs nothing on the normalized tables and the normalization cascade keeps working once otel is restricted.
• Init/readiness probes use EXISTS TABLE on a table the pod's user is granted (privilege-robust, no reliance on system-table row filtering).
• verify-deployment.sh now asserts the whole model on every deploy: each user's grants, the security-critical denials, and that the MVs run as schema_owner.

Key decisions

• Config <grants>, not SQL RBAC — declarative and in-chart. A throwaway verification spike confirmed config grants actually restrict config-defined users, and that access_management cannot be combined with <grants> — so admin uses GRANT ALL … WITH GRANT OPTION, which already carries the access-management privilege group.
• MVs pinned to a stable DEFINER — on this ClickHouse version an MV defaults to DEFINER = its creator, so pre-existing MVs are implicitly owned by otel; reparenting them is required (not just hardening) before otel can be locked down.
• otel lock-down is flag-gated (restrictGrants, default off) so it ships inert and is enabled only after external readers have moved to monte_carlo.
• admin defaults to loopback-only — the superuser is never on a network path; using it requires pod access.
• Append-only model confirmed — no caller mutates rows, so only the schema Job needs anything beyond SELECT/INSERT.

Notes

• Each password-backed user has its own ExternalSecret; schema_owner / llm_worker / monte_carlo are always provisioned, and the README install walkthroughs wire all of them.
• Public platform docs need a matching update (tracked separately), notably switching the documented agent connection to monte_carlo once that cutover happens.

🤖 Generated with Claude Code

---

• Ran /code-review

[toffer]

Code Review — 19 findings (18 fixed, 1 skipped) · validated in k3d

Reviewed: 2026-06-27 UTC | Reviewers: security, devdocs, architecture, correctness, testing, coordinator | Scope: full PR

#	Severity	Finding	Status
F1	ISSUE	expect_denied case-sensitive ACCESS_DENIED grep — fragile across CH versions	✅ Case-insensitive, broadened pattern
F2	ISSUE	readonly_user forbid needle "INSERT" too loose; no runtime check	✅ Tightened to GRANT INSERT + runtime write denial
F3	ISSUE	monte_carlo telemetry-write denial was grant-text only	✅ Now a runtime INSERT attempt
F4	ISSUE	schema_owner "no access management" property unverified	✅ Added access-mgmt + CREATE USER denials
F5	ISSUE	otel restrict state reported, never asserted	✅ Asserts via VERIFY_OTEL_RESTRICTED
F6	ISSUE	MV DEFINER grep may miss CH identifier quoting	✅ Quoting/case-tolerant grep (hedge)
F7	ISSUE	schema_owner ALTER ON system.* + global CREATE DATABASE over-broad	✅ Enumerated 7 system-log tables; documented why CREATE DATABASE stays global
F8	ISSUE	EXISTS TABLE probe may need system.tables SELECT	⏭️ Skipped — confirmed non-issue in k3d (works for llm_worker AND INSERT-only otel)
F9	ISSUE	README user-model table misstated schema_owner/otel reads + otel_metrics	✅ Table corrected + runtime-created footnote
F10	SUGGESTION	Passwords passed as --password CLI args (process list/audit leak)	✅ Via env CLICKHOUSE_PASSWORD
F11	SUGGESTION	otel values shape inconsistent with other users	✅ Renamed to nested clickhouse.otel.* (breaking, no aliases)
F12	SUGGESTION	No upgrade guide for existing installs	✅ Added 1.x→2.0.0 upgrade section
F13	SUGGESTION	admin needle misses WITH GRANT OPTION	✅ Fixed — split into two substring checks (first attempt false-FAILed in k3d: SHOW GRANTS interpolates TO admin)
F14	SUGGESTION	Undifferentiated missing-secret message	✅ Names the specific missing user
F15	SUGGESTION	Helper nindent contract / required paths undocumented	✅ Documented
F16	SUGGESTION	User defs split across 3 files, no cross-ref	✅ Cross-ref comments + aligned ordering
F17	SUGGESTION	Queue-user grant/denial coverage gaps	✅ Added checks
F18	NIT	--- separator placement in range loop	✅ Moved into the ExternalSecret helper
F19	ISSUE	Chart.yaml not bumped for a breaking change	✅ Bumped to 2.0.0

👉 Fix commits: 1afe264 · bc398a1 (k3d-found F13 correction)

k3d validation (2.0.0 chart, full 1.x→2.0.0 upgrade path): helm lint clean; ESO synced all 6 per-user secrets; default superuser removed; 0014 MV-DEFINER reparent confirmed (all 3 MVs DEFINER = schema_owner); all user-model assertions pass pre-cutover and post-cutover (restrictGrants=true + VERIFY_OTEL_RESTRICTED=true); llm-worker + INSERT-only otel collector init EXISTS TABLE probes both pass.