Summary
On 2026-09-01 the bleeding-edge host was effectively down for all 101 identities for over two hours, while reporting Application started, passing its Docker healthcheck (Up 2 hours (healthy)), and serving traffic.
The underlying cause was a schema/code version skew, and the error message for it is good. The problem is that the failure is loud per tenant and completely silent in aggregate, so nothing about the host's observable state said "this box is serving nothing".
What happened
The running image was odin-core-internal:main__main. The tenant databases had been migrated to 202608271000, a migration that exists only on step-1-circle-and-app-registration-tables and is not on main. So every tenant hit the guard in AbstractMigrator.cs:90:
Error loading registration "...": Current database version 202608271000 is higher than
the latest known migration version 202608040942. This likely indicates that the database
was migrated with a newer version of the software.
LoadRegistrations (FileSystemIdentityRegistry.cs:452) catches per registration, logs ERR, and continues. That catch is correct in itself — one broken tenant should not take down the host. But the throw lands before CacheCertificateAsync (FileSystemIdentityRegistry.cs:516), so no tenant certificate was ever cached, and the TLS listener had nothing to present for any identity domain:
|
before restart |
after |
Loaded Identity |
— |
0 |
Certificate loaded for |
98 |
0 |
Error loading registration |
— |
100 |
Externally every identity accepted TCP on 443 and returned no peer certificate available. provisioning.demo.dominion.id kept serving a valid cert from the same listener (system cert, different path), which made the box look half-alive rather than broken.
Why it was hard to spot
- 100 identical ERRs, one per tenant, in a 29 MB daily log — no aggregate line.
- Nothing ever logs "loaded 0 of 101 registrations".
Application started fires regardless.- The healthcheck is
curl -fsS http://127.0.0.1/.well-known/acme-challenge/ping, a system endpoint that answers fine with zero tenants loaded, so Docker reported healthy.
StartupVerificationBackgroundService checks DNS and email infrastructure but not whether any tenant loaded. Its verdict was "1 email issue" (an unrelated MTA-STS SSL failure), which actively pointed away from the real problem.
Requested changes
1. Aggregate result line at the end of LoadRegistrations
Count successes and failures in the loop and log a summary after it — "Loaded {n} of {total} registrations" — at ERR when n == 0, WRN when some failed, INF otherwise. One line, and this incident would have been obvious immediately.
2. Fail fast on host-wide version skew
A MigrationException for version skew is identical for every tenant and is never recoverable by continuing — categorically different from one tenant with a corrupt directory. Check the schema version once, before the tenant loop, and refuse to start with the existing message. The message already tells the operator exactly what to do; it just needs to stop the host instead of scrolling past 100 times.
Keeping the per-tenant catch for genuinely per-tenant faults is still right — the ask is to separate host-wide conditions from per-tenant ones.
Out of scope here
Making the healthcheck reflect that at least one registration loaded would be the third fix, but it touches the Dockerfile/ops side.
How this was diagnosed
Read from the live box: /identity-host/log/app-20260901.log, docker inspect identity-host, and openssl s_client probes against tenant domains. Branch attribution (202608271000 present only on step-1-circle-and-app-registration-tables, 35 commits ahead of main, unmerged) was verified with git ls-tree across all remote branches.
The specific trigger on bleeding was a restored Postgres snapshot taken from an environment running that branch — that part is inferred from the version plus Todd's account of restoring Postgres, not verified against the snapshot itself. The code-level findings above do not depend on it.
Summary
On 2026-09-01 the bleeding-edge host was effectively down for all 101 identities for over two hours, while reporting
Application started, passing its Docker healthcheck (Up 2 hours (healthy)), and serving traffic.The underlying cause was a schema/code version skew, and the error message for it is good. The problem is that the failure is loud per tenant and completely silent in aggregate, so nothing about the host's observable state said "this box is serving nothing".
What happened
The running image was
odin-core-internal:main__main. The tenant databases had been migrated to202608271000, a migration that exists only onstep-1-circle-and-app-registration-tablesand is not onmain. So every tenant hit the guard inAbstractMigrator.cs:90:LoadRegistrations(FileSystemIdentityRegistry.cs:452) catches per registration, logs ERR, and continues. That catch is correct in itself — one broken tenant should not take down the host. But the throw lands beforeCacheCertificateAsync(FileSystemIdentityRegistry.cs:516), so no tenant certificate was ever cached, and the TLS listener had nothing to present for any identity domain:Loaded IdentityCertificate loaded forError loading registrationExternally every identity accepted TCP on 443 and returned
no peer certificate available.provisioning.demo.dominion.idkept serving a valid cert from the same listener (system cert, different path), which made the box look half-alive rather than broken.Why it was hard to spot
Application startedfires regardless.curl -fsS http://127.0.0.1/.well-known/acme-challenge/ping, a system endpoint that answers fine with zero tenants loaded, so Docker reported healthy.StartupVerificationBackgroundServicechecks DNS and email infrastructure but not whether any tenant loaded. Its verdict was "1 email issue" (an unrelated MTA-STS SSL failure), which actively pointed away from the real problem.Requested changes
1. Aggregate result line at the end of
LoadRegistrationsCount successes and failures in the loop and log a summary after it —
"Loaded {n} of {total} registrations"— at ERR whenn == 0, WRN when some failed, INF otherwise. One line, and this incident would have been obvious immediately.2. Fail fast on host-wide version skew
A
MigrationExceptionfor version skew is identical for every tenant and is never recoverable by continuing — categorically different from one tenant with a corrupt directory. Check the schema version once, before the tenant loop, and refuse to start with the existing message. The message already tells the operator exactly what to do; it just needs to stop the host instead of scrolling past 100 times.Keeping the per-tenant catch for genuinely per-tenant faults is still right — the ask is to separate host-wide conditions from per-tenant ones.
Out of scope here
Making the healthcheck reflect that at least one registration loaded would be the third fix, but it touches the Dockerfile/ops side.
How this was diagnosed
Read from the live box:
/identity-host/log/app-20260901.log,docker inspect identity-host, andopenssl s_clientprobes against tenant domains. Branch attribution (202608271000present only onstep-1-circle-and-app-registration-tables, 35 commits ahead ofmain, unmerged) was verified withgit ls-treeacross all remote branches.The specific trigger on bleeding was a restored Postgres snapshot taken from an environment running that branch — that part is inferred from the version plus Todd's account of restoring Postgres, not verified against the snapshot itself. The code-level findings above do not depend on it.