Skip to content

feat(core): add encrypted projection snapshot canary - #1488

Merged
hmans merged 20 commits into
mainfrom
hmans/es-architecture-cleanup-v2
Jul 14, 2026
Merged

feat(core): add encrypted projection snapshot canary#1488
hmans merged 20 commits into
mainfrom
hmans/es-architecture-cleanup-v2

Conversation

@hmans

@hmans hmans commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Why

Projection startup cost grows with retained EVT history, but adding snapshots
introduces compatibility, confidentiality, storage, and multi-replica failure
modes. This PR adds a disabled-by-default Thread projection canary so we can
validate that machinery without expiring EVT or making snapshots part of
recovery correctness.

Closes #1496. Advances #1491 and #1034. A Thread-only snapshot intentionally
does not reduce total startup wall time while all projections share one full
EVT replay fanout.

What changed

  • add core.projection_snapshots configuration and the
    CHATTO_CORE_PROJECTION_SNAPSHOTS override, both off by default; the checked-in
    development cli/chatto.toml enables the canary for local evaluation
  • add projector-level compatibility IDs, validated bootstrap restore,
    skip-through replay, diagnostics, and an atomic state-plus-EVT-sequence
    capture barrier
  • add a private encrypted snapshot repository over SERVER_ASSETS or the
    configured S3 backend using deterministic protobuf, gzip,
    XChaCha20-Poly1305, and an HKDF key derived from core.secret_key
  • publish immutable opaque generations behind an encrypted current/previous
    pointer, with fallback, validation, current/previous referenced retention, and
    failed-publication cleanup
  • add the threads-v1 codec, which excludes message bodies and decrypted PII,
    rebuilds derived indexes, restores transactionally, and derives follow state
    exclusively from durable EVT facts
  • remove the pre-0.5 startup import of legacy thread_follow.* runtime markers;
    those records remain inert in RUNTIME_STATE and are not deleted
  • elect one post-boot writer with the existing MEMORY_CACHE lease and treat
    every snapshot load, restore, lease, storage, and cleanup failure as an
    optimization failure that leaves normal EVT-backed service available
  • preserve the EVT incarnation and encrypted snapshot objects through Chatto's
    JetStream backup/restore path, and retry the NATS stream-creation race seen
    when replicas initialize an empty account concurrently
  • synchronize JSM snapshot progress accounting, whose callback may run from
    multiple goroutines, so backup statistics remain race-free
  • record the design in ADR-050 and update architecture, event-sourcing, Thread,
    backup, configuration, documentation rules, and backend-agent guidance

Compatibility and operations

  • EVT remains permanent and authoritative; snapshots cannot enable event
    expiration, truncation, compaction, or archival.
  • Generations are bound to the projection compatibility ID and a versioned,
    opaque EVT incarnation ID stored in stream metadata. Missing, corrupt, stale,
    future, or undecryptable state causes a cold replay. Snapshot loads have a
    15-second startup deadline, and the canary rejects projection payloads larger
    than 64 MiB to bound transient memory.
  • The incarnation ID survives embedded-NATS reconstruction and backup restore,
    but changes when EVT is deleted and recreated. Missing identity metadata is
    deterministically derived once from the stream creation time so concurrent
    replicas converge, then persisted. Snapshot restore and generation use the
    immutable value captured during core setup.
  • As part of the 0.5 projection cutoff, untouched follow choices written only as
    pre-EVT thread_follow.* runtime markers are no longer imported after a
    restart. Current follow/unfollow choices are durable
    ThreadFollowedEvent/ThreadUnfollowedEvent facts in EVT.
  • Transient pointer-read failures abort generation without changing either
    fallback. Crash, stale-writer, and key-rotation orphan cleanup remains tracked
    in feat(core): productionize snapshot lifecycle and operations #1494 before broader production use.
  • Changing core.secret_key invalidates snapshots but does not lose domain
    data. NATS-backed snapshots are included in chatto backup; S3-backed
    snapshots follow the deployment's S3 backup policy.
  • The new protobuf is an internal persisted cache format, not public API
    surface. It is additive and snapshots using unknown compatibility IDs are
    disposable.

Test plan

  • mise test-cli (pass after merging current main)
  • mise x -- go test -race ./internal/projectionsnapshot ./internal/events ./internal/core -run 'Snapshot|Envelope|Repository|ThreadProjectionSnapshot|EVTStream' (pass)
  • repeated lifecycle/concurrency tests require zero Thread replay after restore,
    no replacement generation, rejection after EVT recreation, and convergent
    concurrent initialization
  • TestBackupRestoreRoundTrip preserves the EVT incarnation metadata and
    private projection snapshot object bytes on a fresh NATS server
  • the hardened backup round trip passes repeatedly under Go's race detector
  • mise lint (pass: Svelte check and ESLint)
  • mise license-check (pass)
  • mise codegen-proto (pass; generated tree clean)
  • mise x -- pnpm build in apps/docs-website (pass)
  • mise x -- go test ./internal/core -run '^$' -bench BenchmarkThreadProjectionSnapshotCodec -benchtime=1x (pass at 1,000 and 10,000 messages)

mise x -- go vet ./... still reports two pre-existing protobuf lock-copy
warnings in internal/core/config_manager_test.go and
internal/core/voice_test.go; this branch does not touch either site.

hmans added 9 commits July 13, 2026 17:31
…-cleanup-v2

* origin/main:
  fix(frontend): preserve expanded timeline groups (#1483)
  feat(notifications): add test push action (#1480)
  fix(frontend): restore branded PWA install icons (#1478)
  fix: forward-port 0.4.9 changes (#1477)
  docs(livekit): document outbound STUN requirements (#1468)
…-cleanup-v2

* origin/main:
  feat(docs): publish stable and development channels (#1490)
  fix(frontend): enforce mono voice call audio (#1489)
  fix(ci): time out media environment setup (#1486)
@hmans hmans changed the title feat(core): add experimental projection snapshot canary feat(core): add encrypted projection snapshot canary Jul 13, 2026
hmans added 11 commits July 13, 2026 20:37
…-cleanup-v2

* origin/main:
  fix(security): restrict public server assets (#1499)
  test(frontend): strengthen design system guardrails (#1516)
  refactor(frontend): codify the design system (#1514)
  fix(config): configure API response compression (#1502)
  fix(frontend): hide deleted users from membership events (#1504)
  fix(dockercompose): use muxed LiveKit UDP port (#1503)
  feat(shields): add Shields.io community badges (#1467)
…-cleanup-v2

* origin/main:
  perf(realtime): centralize myEvents fanout (#1513)

# Conflicts:
#	docs/adr/INDEX.md
@hmans
hmans merged commit 835737e into main Jul 14, 2026
11 checks passed
@hmans
hmans deleted the hmans/es-architecture-cleanup-v2 branch July 14, 2026 14:40
hmans added a commit that referenced this pull request Jul 15, 2026
…e-mapping

* origin/main: (35 commits)
  feat(frontend): add inline message timestamps (#1558)
  docs(architecture): improve inventory readability (#1555)
  perf(connect): reuse DEKs across request hydration (#1554)
  docs(architecture): split runtime inventory by category (#1552)
  fix(core): decrypt projected user PII on demand (#1551)
  fix(frontend): avoid blank line when enabling rich mode (#1548)
  feat(rooms): add joinable room previews (#1546)
  feat(core): bound projection snapshot cleanup (#1538)
  fix(frontend): restore subtle design colors (#1545)
  fix(core): preserve cleared RBAC defaults (#1543)
  fix(pwa): use server name for installed app (#1542)
  fix(frontend): show pending permission updates (#1540)
  fix(frontend): collapse invisible markdown spacing (#1539)
  feat(frontend): expand lazy-loaded locale support (#1537)
  refactor(frontend): modernize and align the design system (#1518)
  feat(core): add encrypted projection snapshot canary (#1488)
  fix(frontend): allow app-wide mobile sidebar swipes (#1534)
  ci: add remote server compatibility smoke test (#1531)
  feat(frontend): add regional English locales (#1532)
  fix(frontend): support remote-only sessions (#1530)
  ...

# Conflicts:
#	cli/internal/core/external_identities.go
#	docs/ARCHITECTURE.md
#	docs/adr/INDEX.md
#	docs/fdr/FDR-001-roles-and-permissions.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(core): land and evaluate the encrypted Thread snapshot canary

1 participant