Skip to content

chore(monorepo): audit + targeted fixes for production readiness - #3

Merged
ethanj merged 6 commits into
mainfrom
audit/monorepo-state
May 18, 2026
Merged

chore(monorepo): audit + targeted fixes for production readiness#3
ethanj merged 6 commits into
mainfrom
audit/monorepo-state

Conversation

@moralespanitz

Copy link
Copy Markdown
Contributor

Summary

Audit on audit/monorepo-state. Lands four narrow fixes plus the infrastructure to make the monorepo build reproducibly from a fresh checkout. Every AGENTS.md gate passes. Full audit in docs/findings/monorepo-audit.md.

Fixes

  • zod v4 schema compatibility. Adds .optional() to silently-coercing z.unknown() helpers across common.ts, memories.ts, documents.ts, agents.ts. Without this, 165 route + schema tests fail with expected: nonoptional under the repo's pinned zod@^4.3.6.
  • Search ranking. memory-search.ts:463 — current-state-query packaging sorts by ranking_score instead of raw score, so the similarity floor actually gates response order. Addresses the "Neovim ranks Phase 4 cutover prep: release queue, docs cleanup, AUDN-SC rename #4" regression. See docs/findings/search-ranking-bias.md.
  • SDK transport errors. SDK now throws a typed NetworkError carrying URL + errno (ECONNREFUSED / AbortError / etc.) instead of bare "fetch failed". CLI hooks now surface actionable messages when core is unreachable.
  • Monorepo build. Rewrites packages/core/Dockerfile to the canonical turbo prune + pnpm deploy --prod pattern; build context shifts to the workspace root via packages/core/docker-compose.yml. Adds root .dockerignore, .npmrc with link-workspace-packages=deep + prefer-workspace-packages=true so consumers resolve @atomicmemory/sdk to the workspace package locally. The prepublishOnly guards still reject workspace: ranges, so published builds remain registry-pinned.

Cleanup (no behavior change)

Trim multi-paragraph AI-generated file headers and stale @example blocks from packages/sdk/src/core/error-handling/*, core/events.ts, core/runtime-config.ts, utils/environment.ts, and utils/validators/primitive.ts. Public surface is documented in packages/sdk/README.md; file headers now state purpose in one line each per AGENTS.md.

Findings (engineering follow-ups, not patched here)

  • docs/findings/concurrent-ingest-race.md — Bug feat(monorepo): public foundation #1 from the integration validation report. Maps the silent-write race in the ingest pipeline + records the recommended fix (unique (user_id, content_hash) index + ON CONFLICT + response-shape change). Multi-day work; not a single-session patch.
  • docs/findings/search-ranking-bias.md — full root cause for the ranking regression. Partial mitigation lands in this PR; two pipeline-internal sorts still use raw score and are flagged for follow-up.

Validation

Surface Result
pnpm run build 10/10, 9.4s
pnpm run typecheck 10/10, 3.0s
pnpm run lint 10/10, 1.4s
pnpm run test (non-core lane) 14/14, 6.6s
pnpm run package-metadata OK
pnpm run repo-hygiene OK
pnpm run security-compliance OK
pnpm run docs-contract OK
pnpm run public-integration-smoke OK (11 package-protocol rows)
pnpm run pack-dry-run OK (12 publishable packages)
pnpm --filter @atomicmemory/core test 3117 pass / 10 skipped
pnpm --filter @atomicmemory/core test:docker-smoke 11/11
pnpm --filter @atomicmemory/core code-health (fallow) 0 dead anything across 119,857 LOC
Workspace Tests
@atomicmemory/core 3117 pass
@atomicmemory/sdk 568 pass
@atomicmemory/cli 377 pass
@atomicmemory/mcp-server 24 pass
@atomicmemory/langchain 19 pass
@atomicmemory/langgraph 17 pass
@atomicmemory/mastra 17 pass
@atomicmemory/openai-agents 11 pass
@atomicmemory/vercel-ai 28 pass
@atomicmemory/openclaw-plugin 3 pass

Grand total: ~4,181 automated tests, 0 failing.

Test plan

  • CI runs ci:affected + ci:public-smoke + ci:docs-contract + ci:pack-dry-run against this branch and passes.
  • Reviewer reads docs/findings/monorepo-audit.md and confirms verdict before merge.
  • (Optional) Reviewer runs pnpm --filter @atomicmemory/core test:docker-smoke locally to spot-check the Dockerfile rewrite.
  • Open follow-up issues for the two items in docs/findings/concurrent-ingest-race.md and the remaining pipeline-internal sorts in docs/findings/search-ranking-bias.md.

Branch shape

26 files changed, +541 / -280. New: .dockerignore, .npmrc, docs/findings/*.md.

Audit on `audit/monorepo-state`. Lands four narrow fixes plus the
infrastructure to make the monorepo build reproducibly from a fresh
checkout. Every AGENTS.md gate passes (build / typecheck / lint / test
/ package-metadata / repo-hygiene / security-compliance / docs-contract
/ public-integration-smoke / pack-dry-run). Full audit report in
`docs/findings/monorepo-audit.md`.

Fixes
-----
- zod v4 schema compatibility — add `.optional()` to silently-coercing
  `z.unknown()` helpers (`common.ts` Workspace/Agent/Visibility/AgentScope
  /IsoTimestamp/OptionalBodyString, `memories.ts` OptionalBoolean/Session
  /OptionalQuery/OptionalUuidQuery/source_memory_ids, `documents.ts`
  OptionalPositiveBigInt/OptionalIsoTimestamp/PointerOnlyStorageMode,
  `agents.ts` display_name). Without this, 165 route + schema tests fail
  with `expected: nonoptional`.
- search ranking — current-state-query packaging now sorts by
  `ranking_score` instead of raw `score` (`memory-search.ts:463`) so the
  similarity floor actually gates response order. Addresses the "Neovim
  ranks #4" regression. See `docs/findings/search-ranking-bias.md`.
- SDK transport errors — SDK throws a typed `NetworkError` carrying URL
  and errno (`ECONNREFUSED` / `AbortError` / etc.) instead of bare
  "fetch failed". Hooks and the CLI now surface actionable messages when
  core is unreachable.
- monorepo build — rewrite `packages/core/Dockerfile` to the canonical
  `turbo prune` + `pnpm deploy --prod` pattern; build context shifts to
  the workspace root via `packages/core/docker-compose.yml`. Adds root
  `.dockerignore`, `.npmrc` with `link-workspace-packages=deep` +
  `prefer-workspace-packages=true` so consumers resolve
  `@atomicmemory/sdk` to the workspace package locally (the registry pin
  is preserved for publish via the existing `prepublishOnly` guards).
- `.gitignore` — extends env-ignore pattern to allow all `*.example`
  templates, adds smoke-test scratch, `out/` (turbo prune), and
  `data/raw-storage/` (local FS storage default).

Cleanup (no behavior change)
----------------------------
- Trim multi-paragraph AI-generated file headers and stale `@example`
  blocks from `packages/sdk/src/core/error-handling/*`, `core/events.ts`,
  `core/runtime-config.ts`, `utils/environment.ts`, and
  `utils/validators/primitive.ts`. Public surface is documented in
  `packages/sdk/README.md`; the file headers now state purpose in one
  line each per AGENTS.md.

Findings (engineering follow-ups, not patched here)
---------------------------------------------------
- `docs/findings/concurrent-ingest-race.md` — Bug #1 from the validation
  report. Maps the silent-write race in the ingest pipeline + records
  the recommended fix (unique `(user_id, content_hash)` index +
  `ON CONFLICT` + response-shape change). Multi-day work; not a
  single-session patch.
- `docs/findings/search-ranking-bias.md` — root cause for the ranking
  regression. Partial mitigation in this commit; two pipeline-internal
  sorts still use raw `score` and are flagged.

Validation
----------
- core: 3117 vitest pass / 10 skipped
- sdk: 568 pass / 1 skipped
- cli: 377 pass / 4 skipped
- mcp-server: 24 pass
- adapters/{langchain,langgraph,mastra,openai-agents,vercel-ai}:
  19/17/17/11/28 pass
- plugins/openclaw: 3 pass
- `test:docker-smoke`: 11/11
- `public-integration-smoke`: 11 package-protocol rows
- fallow code-health on core: 0 dead files / 0 dead exports / 0 unused
  deps / 0 hotspots across 119,857 LOC
…cleanup

CI surfaced two issues on PR #3:

1. `langgraph#lint` (tsc --noEmit) failed because the SDK now exported
   two classes named `NetworkError` — the existing one in
   `core/error-handling/errors.ts` and the new memory-layer one I added
   in `memory/errors.ts`. The top-level `src/index.ts` re-exports both,
   producing a duplicate export. Rename the new class to
   `MemoryTransportError` (different inheritance tree, different
   responsibility — naming is now unambiguous).

2. `sdk#code-health` (fallow audit) failed because the SDK comment
   cleanup shifted line numbers across `core/error-handling/*`,
   `core/events.ts`, `core/runtime-config.ts`, and
   `utils/{environment.ts,validators/primitive.ts}`. Those file
   headers existed on origin/main and contained stale
   `// fallow-ignore-next-line unused-class-member` suppressions whose
   line targets shifted when the headers shrank. Revert all 10 comment-
   cleanup files to their origin/main state. The Bug #5 fix (typed
   transport error in `memory/errors.ts` + `memory/shared/http-client.ts`)
   stays — that's the substantive change.

After this commit, fallow's audit scope drops the reverted files (they
match origin/main exactly), and the only SDK files in scope are the two
behaviorally-changed ones for the transport-error fix.
@ethanj
ethanj marked this pull request as ready for review May 18, 2026 18:46
ethanj and others added 4 commits May 18, 2026 11:57
Make Turbo build dependency packages before lint so workspace-linked packages expose their generated dist type declarations during affected CI.

Keep the workspace-link lockfile entries from the PR while restoring unrelated transitive resolutions to the existing frozen versions.
Keep the public docs tree user-facing by removing migration provenance and cutover artifacts from docs/migration.

Drop the migration inventory CI hook that depended on those internal artifacts and add a repo-hygiene guard so docs/migration cannot be reintroduced accidentally.
Move remaining audit finding notes out of the public repository and remove the now-empty docs tree, plus stale layout and CODEOWNERS references.
Closes the two pipeline-internal sites identified during the original
ranking audit but not patched in 8871d7d:

- `search-pipeline.ts:1126` — RRF expansion merge (entity / temporal /
  linked memories). Was sorted by raw `score`, which lets the
  unconditional `(α·sim + β·importance + γ·recency) · trust` formula
  win even when similarity is below the `ranking-min-similarity` floor.
- `search-pipeline.ts:1443` — RRF stage-weighted merge. Same issue.

Both now sort by `(b.ranking_score ?? b.score) - (a.ranking_score ?? a.score)`,
matching the final-packaging fix already in `memory-search.ts:463`. The
similarity floor now gates every retrieval-pipeline ordering decision,
not just the response-packaging step.

All 3117 core vitest tests still pass — no behavior change for queries
where similarity is above the floor (ranking_score equals raw score in
that case).
@ethanj
ethanj merged commit 0ef8e09 into main May 18, 2026
9 checks passed
@ethanj
ethanj deleted the audit/monorepo-state branch May 18, 2026 20:01
ethanj added a commit that referenced this pull request May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants