architecture.md

yaggo-brain architecture

Español: es/architecture.md. This document reflects the current implementation (through the sandbox/security and agent-catalog work).

Monorepo layout:

• apps/web — Next.js panel (Explorer, Graph, Knowledge, Wiki, Docs, Vectors, Costs, Cookbook, Hub, Memory, Decisions, Workflow, Terminal, Agents, Lessons, Quality, Review, Skills) + /login + dashboard IDEs
• apps/api — NestJS REST API (+ Hub, Cookbook, Observations SSE, Better Auth at /api/auth/*, POST /api/auth/api-token, GET /api/health/ides)
• apps/worker — BullMQ jobs + MCP HTTP mirror (:37700, JWT on POST /api/tools/* and hooks) + crons
• apps/mcp-server — MCP stdio server (16 search/timeline + 9 get + 6 feedback tools)
• apps/mcp-proxy — HTTP router local worker vs remote VPS MCP (MCP_PROXY_ROUTES, TTL cache for search tools)
• apps/observatory — static public aggregates site
• apps/yaggo-brain-cli — update, contrib, install, doctor, wrap
• apps/sandbox-runner — CLI entry for sandbox commands
• infra/vectors-worker — Python UMAP + HDBSCAN (:37800)
• infra/ragas-worker — Python RAG faithfulness (:37801)
• infra/docling-worker — Python PDF parse (:37802, Docling with pymupdf fallback)
• packages/* — shared libraries (hub, observatory-data, model-router/benchmarks, auth, privacy, sandbox, memory, code-intel)

Security [D-42] (Fase 11)

• Better Auth (packages/auth): email/password, bearer tokens, scopes; mounted at /api/auth/*
• Auth guard (API): global guard; skip /health, /api/auth/*; AUTH_DISABLED=true by default for local dev
• RLS: Postgres policies on workspace-scoped tables via app.workspace_id GUC (withDbContext)
• Privacy: <private> redaction triggers + read-time filter; redacted/private chunks excluded from vector snapshots
• Sandbox: Docker ephemeral runner (infra/sandbox/), seccomp profile, read-only repo bind mount, command guard + audit, SANDBOX_MODE=stub still supported
• Audit: GET /api/projects/:id/audit; blocked commands → audit_log.action = command.blocked_dangerous
• Workspace policy: allow_network_in_sandbox (default false); bridge only with requiresNetwork + audit
• Agent permissions: can_execute_commands enforced on sandbox exec when agentId provided
• CI: .github/workflows/security.yml (gitleaks + license-checker on packages/)

Data flow for /ask

1. Web → API POST /api/projects/:id/ask
2. model-router selects tier/model (optional ROUTER_USE_BENCHMARKS=true → model_profiles)
3. rag hybrid retrieval: pgvector chunks + PageIndex tree scoring (PAGE_INDEX_ENABLED=true)
4. Optional LiteLLM answer; cost-engine records model_calls

Docling PDF ingest (Fase 14)

1. Upload PDF → worker ingest-document
2. If DOCLING_ENABLED=true, POST to docling-worker:37802/parse (Docling or pymupdf fallback)
3. indexDocumentChunks + document_trees PageIndex tree persisted

Memory blocks (Fase 14)

• Local Letta-style blocks in memory_blocks + memory_blocks_index (no external Letta service)
• MCP update_memory_block writes block value; memory_get returns full content
• Default seeds: human, persona, project_context on first update

Decisions + ADRs (Fase 14)

• Table decisions for architectural decisions
• Worker job ingest-adr-decisions imports docs/adr/*.md idempotently by source_path

Community Hub [D-68]

1. CLI yaggo-brain contrib --lesson-id=... → POST /api/hub/contrib
2. API enqueues hub-process-submission job
3. @yaggo-brain/hub generalizes lesson, runs eval gates, writes hub_publications + packages/hub/seed/community-lessons.json
4. Web Hub tab lists hub_submissions status

Model Cookbook [D-70]

1. Seed model_profiles from packages/model-router/benchmarks/seed.json on migrate
2. GET /api/cookbook/recommendations uses detectHardware() stub + profiles
3. Worker ingest-public-benchmarks proposes diff (no overwrite)

Observatory [D-71]

1. @yaggo-brain/observatory-data exports anonymized JSON (k-anonymity ≥ 5)
2. Worker weekly cron export-observatory-snapshot → apps/observatory/public/data/snapshot.json
3. pnpm observatory:build → static site in apps/observatory/dist

Panel extensions (Fase 10d)

• GET /api/observations + SSE /api/observations/stream
• MCP record_observation persists to observations
• Memory tab live stream; Quality metrics (lesson rates, skill pass-rates); A/B compare ask
• GET /api/projects/:id/vectors/status

Panel UX (Fase 15)

New tabs

• Knowledge — aggregated graph (wiki, lessons, decisions, documents) via GET .../knowledge/graph
• Decisions — ADR/decision timeline + Import ADRs job
• Workflow — editable React Flow graph stored in workflows.definition
• Terminal — xterm + one-shot sandbox exec (POST .../sandbox/exec)

Citations flow

1. UI detects type:uuid patterns in answers, observations, lessons (CitationLink)
2. Click opens CitationDrawer → POST /api/citations/resolve → resolveGetByCitedIds (MCP layer 3)
3. Client-side cache in use-citation-resolver

Memory Inspector++

• Split layout: observations SSE + memory blocks editor (PATCH .../memory/blocks/:id)
• 3-layer search: GET .../memory/search (all sources or layer=blocks) + drawer for get layer
• Compress now stub (session.end integration pending)

API additions (Fase 15)

• GET/POST /api/projects/:id/decisions, POST .../decisions/ingest-adrs
• GET/PATCH /api/projects/:id/memory/blocks, GET .../memory/search
• GET/PUT /api/projects/:id/workflows/default
• POST /api/citations/resolve

Sandbox hardening (Fase 16)

1. Terminal or API → POST /api/projects/:id/sandbox/exec with optional agentId
2. guardCommand (regex + chained segments) → on block: recordAudit + 400
3. runInSandbox → Docker with seccomp, cap-drop, read-only rootfs, optional -v repo:/workspace:ro
4. Settings tab → GET .../audit last 50 security events

Build sandbox image: pnpm sandbox:build

Vectors (Fase 6.5)

1. Chunks with embeddings in chunks (pgvector)
2. Worker job build-vector-snapshot POSTs to vectors-worker
3. Snapshot stored in vector_snapshots
4. Web tab Vectors renders UMAP scatter (canvas MVP)

Agent flows (Fase 17)

• Catalog: 14 agents in packages/agent-runtime/src/agents/*.yaml (Zod + permission matrix)
• Readonly runners: architect, documentation, security-agent, reviewer-agent, research-agent
• QA write: generates diff + runs sandbox command (requires can_execute_commands)
• Dispatch: runAgent checks scope, budget (BUDGET_EXCEEDED), and agent.before hooks
• Hooks: bootstrapAgentHooks() on API and worker startup; per-agent subscribed_hooks
• MCP: get_agents lists catalog; invokeTool filters by allowed_tools when agent_id set
• Approve review: worker job apply-review-patch creates git branch + commit
• Reject review: creates project lesson proposed + enqueues quality review
• Agent Board: kanban by run status/review + catalog filters (project/workspace)

Quality loop (Fase 12)

1. Trigger events: failed agent eval, eval suite below threshold, review reject, MCP request_reflexion, session end
2. API/worker enqueues quality-review (cooldown: max 5/workspace/24h via audit_log)
3. @yaggo-brain/quality-agent LangGraph: evaluate → reflect → record lesson → pattern check → optional autoresearch
4. Lessons promote proposed → accepted at 3 distinct evidenceRuns
5. Optional quality_findings emitted; panel GET .../quality/findings

Lifecycle hooks (Fase 12)

• packages/hooks dispatches session.start|end, tool.before|after, prompt.compose
• Worker POST /api/hooks/:event returns 202 and runs handlers async
• Cursor install copies hook scripts to .cursor/hooks/yaggo-brain/ and project hooks.json
• session.end: compress observations → session_summaries + enqueue quality review

Evals

• Datasets in packages/evals/datasets/*.jsonl
• pnpm eval runs 7 suites (incl. uplift-canonical [D-69], rag-faithfulness-canonical)
• pnpm --filter @yaggo-brain/evals ci-gate blocks >5% regression on code-review-canonical

Skills autoresearch

• Catalog under packages/skills/catalog/{skillId}/
• Versions in versions/{semver}/SKILL.md
• ADRs auto-written to docs/adr/ on improvement
• Cron: 0 3 * * * via BullMQ repeatable job

MCP 3-layer (Fase 6 + Fase 14)

• Layer 1 *_search: compact { hits: [{id, title, citedId}], citedIds }
• Layer 2 *_timeline: chronological context around an anchor
• Layer 3 *_get: full content via GetResponse.items[]
• Extra: get_vector_neighbors(chunk_id) for pgvector similarity
• code_search via @yaggo-brain/code-intel (file_nodes + AGE graph)
• Feedback includes update_memory_block

Multi-IDE harness (Fase 13) [D-59]

• yaggo-brain install detects Cursor / Claude Code / Windsurf and delegates to native paths
• Hooks POST to worker POST /api/hooks/:event (session.start|end, prompt.compose, tool.after)
• .yaggo-brain/ide-status.json updated by hooks; API GET /api/health/ides merges DB + file
• mcp-proxy routes tool calls per projectId (local worker vs remote VPS)
• POST /api/auth/api-token issues scoped bearer tokens for MCP/hooks
• Plugin manifests: .cursor-plugin/, .claude-plugin/