Skip to content

FIXES_AND_LEARNINGS.md

Latest commit

 

History

History
420 lines (297 loc) · 21.8 KB

FIXES_AND_LEARNINGS.md

File metadata and controls

420 lines (297 loc) · 21.8 KB
Raw

MemoryWeave — Fixes & Learnings

Settings / Reports refetched Coral on every navigation

Date: 2026-05-29 Phase: Phase 6 — Production UX Severity: Medium

What Happened

On production, Settings stayed on “Coral CLI Checking…” and Reports showed skeleton loaders for a long time. Leaving either page and returning triggered the same wait again, as if the app reloaded Coral from scratch every visit.

Root Cause

React Router unmounts page components on navigation. SettingsPage, ReportsPage, and McpIntegrationSection each used local useState + useEffect to call GET /coral-schema, GET /coral-report, and GET /coral-mcp-config on every mount. /coral-report runs multiple Coral SQL subprocesses on Render and can take many seconds; there was no client-side cache, so revisiting the page always looked like a cold load.

How It Was Fixed

Moved Coral API responses into Zustand (appStore.js) with a 5-minute TTL (CORAL_CACHE_TTL_MS, aligned with the Settings “Cache TTL” stat card):

  • fetchCoralSchema, fetchCoralReport, fetchCoralMcpConfig skip the network when cache is fresh.
  • Stale-while-revalidate: if data exists but TTL expired, the UI keeps showing cached rows while a background refetch runs (no full-page spinner).
  • Loading spinners only when there is no cached data yet (status === 'loading' && !coralReport).

Files: frontend/src/stores/appStore.js, SettingsPage.jsx, ReportsPage.jsx, McpIntegrationSection.jsx.

What I Learned

Route-level useEffect fetch is wrong for expensive, slow-changing data. Cache in global state (or React Query) when users hop between dashboard tabs. Match TTL to product copy (300s on Settings) so behavior is explainable to judges.

Relevant for Interview

Demonstrates SPA performance debugging: distinguish slow backend from unnecessary refetch, and fix with session cache without changing API contracts.

Coral GitHub source — live API vs JSONL fallback

Date: 2026-05-29 Phase: Phase 6 — Coral hackathon polish Severity: Medium

What Happened

Judges expect a live external API in the Coral story, not only local JSONL. Coral 0.4.1 requires GITHUB_TOKEN for the bundled github source (no unauthenticated install).

Root Cause

Coral's github bundled source marks GITHUB_TOKEN as required at coral source add time. Public GitHub REST allows unauthenticated reads, but Coral's install path does not.

How It Was Fixed

install_sources.sh tries coral source add github when GITHUB_TOKEN is set and smoke-tests github.issues for Venkat-Kolasani/MemoryWeave. On failure or missing token, queries use memoryweave_demo.github_issues from coral/data/github_issues.jsonl. Mode is written to $CORAL_CONFIG_DIR/github_mode and exposed on GET /coral-schema as github_mode.

What I Learned

Ship a file fallback for demos and CI; use the live API when credentials exist. Document both paths so judges can reproduce without blocking on PAT setup.

Relevant for Interview

Shows pragmatic integration design: sponsor feature (live API) + reliable fallback for hackathon demos.

Frontend fetch failures from missing Vercel CORS origin

Date: 2026-05-29 Phase: Phase 6 — Production verification Severity: Critical

What Happened

Direct API checks passed, but the Vercel frontend showed blank Risk cards, Reports Failed to fetch, and Settings Coral CLI Not Found.

Root Cause

Render ALLOWED_ORIGINS did not include https://memory-weave-ai.vercel.app, so browser preflight requests returned 400 Disallowed CORS origin. Server-to-server curl checks were misleading because they bypass CORS.

How It Was Fixed

The backend now always includes the deployed Vercel origin and localhost in its CORS allowlist, then merges any extra ALLOWED_ORIGINS from Render. The deploy docs now use the exact Vercel URL.

What I Learned

Production verification must test browser-origin CORS preflight, not only direct endpoint responses.

Relevant for Interview

Shows full-stack debugging across frontend browser behavior, backend middleware, and deployment environment.

Render Coral — Aura HTTP API blocked by administrative rules

Date: 2026-05-29 Phase: Phase 6 — Coral integration Severity: Critical

What Happened

Even after setting the database path to /db/neo4j/tx/commit, production Coral graph queries failed with 403 Forbidden from Neo4j Aura's HTTP transactional endpoint.

Root Cause

Aura accepted Bolt connections for the app, but rejected the HTTP transactional endpoint used by Coral's HTTP source. Coral schema discovery still passed because the local demo JSONL source worked, masking graph-source failures until /health/deep and /coral-query.

How It Was Fixed

Changed memoryweave_graph from a Neo4j HTTP source to packaged JSONL graph snapshots (knowledge_nodes.jsonl, knowledge_edges.jsonl). The app UI still reads the live Neo4j graph; Coral SQL now has a reliable production read layer for the hackathon demo.

What I Learned

When a third-party API blocks a transport, preserve the product path with a stable snapshot source rather than burning demo time on infrastructure edge cases.

Relevant for Interview

Shows pragmatic incident response: isolate the failing integration boundary, keep user-facing behavior intact, and document the tradeoff.

Render Coral — Aura HTTP database path 403

Date: 2026-05-28 Phase: Phase 6 — Coral integration Severity: Critical

What Happened

Production /coral-query failed with Source request was rejected (403) against https://7459e09d.databases.neo4j.io/db/7459e09d/tx/commit.

Root Cause

Coral uses Neo4j's HTTP transactional endpoint, where Aura expects the database path to be /db/neo4j/tx/commit. The install script defaulted NEO4J_DATABASE to the Aura username/instance id, producing /db/<instance-id>/tx/commit.

How It Was Fixed

Changed the Coral install script default to NEO4J_DATABASE=neo4j and updated /health/deep to smoke-test memoryweave_graph.knowledge_nodes instead of only the local demo JSONL source.

What I Learned

Aura's Bolt username and HTTP transactional database path are different concerns; a green CLI/schema check does not prove graph-source queryability.

Relevant for Interview

Good example of testing the actual integration path, not only dependency availability.

Render Docker — Coral binary required GLIBC_2.39

Date: 2026-05-28 Phase: Phase 6 — Coral integration Severity: Critical

What Happened

Docker build installed Coral successfully, but coral --version failed with /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.39' not found.

Root Cause

The image used python:3.11-slim-bookworm, which is Debian Bookworm with glibc 2.36. Coral v0.4.1's Linux binary requires glibc 2.39 or newer.

How It Was Fixed

Changed the backend Docker base image to python:3.11-slim-trixie, which provides a new enough glibc for Coral.

What I Learned

Binary CLI tools can impose OS libc constraints independent of Python dependencies; verifying the binary during Docker build caught this before runtime.

Relevant for Interview

Good example of diagnosing native binary compatibility issues in containerized deployments.

Render Docker — Coral install 404 (wrong CORAL_VERSION tag)

Date: 2026-05-28 Phase: Phase 6 — Coral integration Severity: Critical

What Happened

Docker build failed at RUN curl ... install.sh with curl: (22) The requested URL returned error: 404 after Installing Coral 0.4.1 for x86_64-unknown-linux-gnu.

Root Cause

CORAL_VERSION=0.4.1 without the v prefix. GitHub assets live under /releases/download/v0.4.1/, not /releases/download/0.4.1/.

How It Was Fixed

Set ARG CORAL_VERSION=v0.4.1 in backend/Dockerfile.

What I Learned

Always match the exact GitHub release tag string when pinning binary downloads.

Relevant for Interview

Debugging container builds from opaque install scripts.

Render Docker — Coral CLI not on PATH at runtime

Date: 2026-05-28 Phase: Phase 6 — Coral integration Severity: High

What Happened

Render deploy logs showed [coral] CLI not found and [coral] setup error: coral CLI not found despite a successful Docker build.

Root Cause

install.sh defaults to $HOME/.local/bin (/root/.local/bin). The binary was not reliably on PATH at container start, and unpinned CORAL_VERSION could fail silently against GitHub API rate limits during build.

How It Was Fixed

  • Dockerfile: CORAL_INSTALL_DIR=/usr/local/bin, pin CORAL_VERSION=v0.4.1 (GitHub tag prefix), run coral --version in build (fails image if install breaks).
  • start.sh: prepend /usr/local/bin to PATH; optional runtime install fallback.

What I Learned

Install CLI tools to a global path in Docker (/usr/local/bin) and verify in the same RUN layer.

Relevant for Interview

Container PATH and multi-stage/production install gotchas.

/health Coral probe caused probe timeouts (code review)

Date: 2026-05-28 Phase: Phase 6 — Coral integration Severity: High

What Happened

/health instantiated CoralService and ran a full coral sql query on every request. Render liveness probes hit /health frequently; each call spawned subprocesses with up to 45s timeout risk.

Root Cause

Health check mixed “is the process up?” with “can Coral run SQL?”. Module-level OpenAI(**fireworks_client_kwargs()) in coral_query.py also crashed app import when FIREWORKS_API_KEY was unset.

How It Was Fixed

  • Cached Coral status at startup via probe_coral_health(); /health reads get_coral_health_status() only.
  • Added /health/deep for optional SQL smoke test (thread pool).
  • Singleton get_coral_service(); lazy Fireworks client on request.
  • _extract_keyword returns None → OPERATIONAL_CONTEXT_OVERVIEW instead of defaulting to "payment".
  • sanitize_sql_param() before SQL .format(); Settings page neutral loading state.

What I Learned

Liveness endpoints must be O(1) and side-effect free; deep readiness checks belong on a separate route.

Relevant for Interview

Production hygiene: probe design, lazy init, and separating fast vs slow health checks.

Coral CLI install and source spec format (CORAL-01)

Date: 2026-05-28 Phase: Phase 6 — Coral integration Severity: Medium

What Happened

Step 1 used npm install -g @coraldata/coral (404). Step 4 used coral schema --sources (subcommand removed in Coral 0.4.1). The draft sources.yaml bulk sources: list is not valid DSL v3.

Root Cause

Coral ships as a Homebrew cask (withcoral/tap/coral), not an npm package. v0.4.1 uses per-source manifest.yaml files (backend: http|file), validated with coral source lint and installed via coral source add --file. Schema discovery is SELECT … FROM coral.tables.

How It Was Fixed

  • Installed CLI: brew install withcoral/tap/coral → coral 0.4.1
  • Added backend/coral/manifests/memoryweave_graph.yaml (Neo4j HTTP → knowledge_nodes, knowledge_edges)
  • Added backend/coral/manifests/memoryweave_demo.yaml (file/jsonl → slack_messages, incident_reports)
  • Generated backend/coral/data/*.jsonl from demo corpus; backend/coral/install_sources.sh substitutes file:// paths (handles spaces in paths)
  • queries.py uses qualified names (memoryweave_graph.knowledge_nodes, etc.)

What I Learned

Match the tool’s actual packaging and CLI surface before writing integration docs; lint manifests early with coral source lint.

Relevant for Interview

Shows adapting a prompt/spec to real tool constraints without blocking the product narrative.

Coral Integration Branch — Strategy Doc

Date: 2026-05-28 Phase: Phase 6 — Coral integration

What Happened

Started Coral-integration branch to add Coral as the cross-source SQL read layer for MemoryWeave while keeping main stable for the live Vercel + Render demo.

Root Cause

N/A — planned enhancement

How It Was Fixed

N/A — documented approach in docs/CORAL_LOCAL.md; implementation follows on main.

What I Learned

MemoryWeave’s Neo4j graph UI and Patel bus-factor story are the product differentiator; Coral strengthens the Assistant and judge demo with one SQL join across Slack, incidents, and org data — aligned with hackathon scoring for MCP and cross-source queries.

Relevant for Interview

Shows how to extend an existing MVP for multiple submissions without forking the entire architecture: additive read path, branch isolation, clear “why Coral” narrative.

Initial Entry — Log Created

Date: 2026-05-26 Phase: Phase 1

What Happened

Project initialized. This log will track all critical failures and fixes.

Root Cause

N/A

How It Was Fixed

N/A

What I Learned

Documenting failures in real-time is far more accurate than reconstructing from memory.

Relevant for Interview

Shows disciplined engineering habits and self-awareness about failure modes.

Neo4j + ChromaDB Store Implementation

Date: 2026-05-26 Phase: Phase 3

What Happened

Implemented the MemoryWeave data-store layer:

  • backend/services/neo4j_service.py for Neo4j schema creation, graph upserts, relationship linking, frontend graph reads, entity reads, and risk aggregation.
  • backend/data/seed.py for deterministic Acme Corp demo graph seeding.
  • backend/services/chroma_service.py for ChromaDB collection management, document insertion, semantic search, and stats.
  • backend/data/populate_chroma.py for chunking Slack messages, incident markdown, and docs into the memoryweave_knowledge collection.

Root Cause

The frontend and backend were still backed by static fixtures. The project needed real local data-store population scripts before route wiring and RAG query work could proceed.

How It Was Fixed

Added idempotent Neo4j MERGE operations and schema constraints, plus a 21-relationship Acme graph where A. Patel has the highest risk score and the relationship set mirrors the frontend mock graph. Added a Chroma ingestion pipeline using DefaultEmbeddingFunction only, with no sentence-transformers or PyTorch import.

Verification

  • Python source compile checks passed for the four new files.
  • ChromaDB was verified by starting a temporary local server on localhost:8001, indexing 58 chunks, and passing the semantic search assertion for payment service recovery patel.
  • Neo4j live verification passed after local Docker Neo4j was started: seed completed with 15 nodes, 21 relationships, and A. Patel risk score 95. A follow-up query confirmed the live Neo4j edge set matches the frontend mock graph.

What I Learned

Keep demo seed contracts aligned with the frontend visualization shape. The task text mentioned 5 incidents, but also required 15 total nodes and frontend mock-edge compatibility; the implemented Neo4j seed follows the 15-node verification contract while Chroma indexes every incident markdown file available in the corpus.

Live API Wiring — Graph + Ingest (Day 3)

Date: 2026-05-27 Phase: Phase 3 — MVP checkpoint

What Happened

Merged commit 10471f7 wiring backend routes to live data stores:

  • GET /graph reads nodes/edges directly from Neo4j.
  • GET /stats returns live node count from Neo4j (other stat fields still hardcoded).
  • POST /ingest runs a 3-pass Fireworks extraction pipeline (extractor.py + pipeline.py) and persists to Neo4j + ChromaDB.
  • /risk-report and /query remain mock-backed; frontend api.js still uses in-browser mocks.

Root Cause

Data-store services existed but were disconnected from FastAPI routes and the UI — the demo couldn't show live graph or ingestion without this wiring layer.

How It Was Fixed

Added agents/extractor.py (Fireworks JSON extraction) and agents/pipeline.py (chunk → 3-pass → store). Updated graph.py, risk.py (stats only), and ingest.py to call Neo4jService / ChromaService with graceful fallbacks.

Verification

  • TestClient: /graph → 15 nodes, 21 edges; /stats → nodes=15.
  • Neo4j Docker container healthy on localhost:7687.

What I Learned

Wire one vertical slice end-to-end before polishing all endpoints. Graph + ingest first gives a credible MVP demo; query/RAG and frontend fetch switch are the next highest-leverage items.

Relevant for Interview

Demonstrates incremental integration: services → routes → agents → (next) frontend, rather than big-bang wiring at the end.

Live Risk + Hybrid Query Wiring

Date: 2026-05-28 Phase: Phase 3 — risk + query checkpoint

What Happened

Implemented the two remaining live backend endpoints:

  • GET /risk-report now computes bus-factor risk from Neo4j with NetworkX in services/risk_scorer.py.
  • POST /query now retrieves semantic context from ChromaDB and graph context from Neo4j in services/retriever.py, then attempts Fireworks structured JSON synthesis.

Root Cause

The risk and assistant routes were still returning static fixtures. The demo needed the backend to prove real graph scoring and grounded retrieval before the frontend mock switch.

How It Was Fixed

Added weighted ownership graph construction from KNOWS, incident RESOLVES/AFFECTS, workflow OWNS/DEPENDS_ON, and system dependency relationships. Added query entity matching with aliases such as "payment service" → "Payment API", graph neighborhood formatting, risk-context injection for broad bus-factor questions, strict JSON response normalization, and a retrieval fallback when Fireworks is unavailable.

Verification

  • Seed passed: 15 nodes, 21 relationships, A. Patel seed risk 95.
  • Chroma population passed: 58 chunks indexed and search assertion passed.
  • /risk-report passed required checks: Payment API owner A. Patel, score 100, at least 2 bottlenecks, Engineering heatmap 86.
  • /query passed required HTTP checks for payment recovery, undocumented expertise, and P-4021 incident questions.
  • Backend project py_compile passed outside .venv.

Open Issues

  • Fireworks rejected the configured model/key locally (404 Model not found/inaccessible; model listing returned 401 Unauthorized), so verified query responses used the retrieval fallback rather than true LLM synthesis.
  • Local pip check fails on grpcio 1.80.0 platform metadata.
  • Local frontend npm run build hangs under Node v25.3.0; retest with Node LTS.

What I Learned

For hackathon demos, retrieval fallbacks are worth the extra hour: they keep the product demonstrable when provider credentials or model access fail, while still making the dependency problem visible in logs and docs.

Fireworks 404 Misdiagnosed as Wrong Model

Date: 2026-05-28 Phase: Phase 4 — pre-deploy verification Severity: High

What Happened

/query and test_fireworks.py returned 404 Model not found for every Llama model ID, including valid Fireworks catalog names.

Root Cause

Two separate issues stacked: (1) backend/.env on disk still held a 12-character placeholder key while the editor showed an unsaved fw_… key; (2) after the real key was saved, the account only had access to serverless models like kimi-k2p5, not llama-v3p1-70b-instruct.

How It Was Fixed

Added validate_fireworks_api_key(), scripts/test_fireworks.py, and switched default model to accounts/fireworks/models/kimi-k2p5 in fireworks_config.py and .env.example.

What I Learned

Fireworks often returns 404 (not 401) for bad keys or inaccessible models — always list models with GET /inference/v1/models before blaming the model string.

Relevant for Interview

Shows systematic API debugging: verify env load → verify auth → list allowed resources → then change config.

localhost:8000 Routed to Wrong Docker App

Date: 2026-05-28 Phase: Phase 4 Severity: High

What Happened

curl http://localhost:8000/query returned {"detail":"Not Found"} while uvicorn logged MemoryWeave as running.

Root Cause

Another Docker container bound *:8000; MemoryWeave uvicorn bound 127.0.0.1:8000. macOS routed localhost to the Docker service, not uvicorn.

How It Was Fixed

Set VITE_API_URL=http://127.0.0.1:8000 in frontend/.env / .env.example and documented the conflict in LOCAL_SETUP.md.

Relevant for Interview

Classic local dev port collision — always compare lsof -i :8000 and test both localhost vs 127.0.0.1.

Phase 5 UI Verification Pass

Date: 2026-05-28 Phase: Phase 5 — polish Severity: Medium

What Happened

Pre-deploy checklist found several UI/UX gaps: graph legend showed total counts during filters, edges used OR-filter logic (orphan edges visible), risk stat cards were hardcoded, assistant warnings rendered inside the badge grid, graph loading skeleton never appeared, and font stack included system-ui.

Root Cause

Phase 1–2 mock-first implementation left placeholder stats and simplified graph filter logic; Zustand seeded MOCK_GRAPH_NODES on boot so graphNodes.length === 0 was never true.

How It Was Fixed

  • Graph: isGraphLoading flag; empty initial graph; edge visibility requires both endpoints to match filter; legend counts scoped to active filter; dynamic subtitle from live node/edge counts.
  • Risk: Stat cards derive from live riskItems; table sorted by score descending.
  • Assistant: Warnings moved to full-width red alert boxes below the related-entity grid; context panel reads knowledgeStats from /stats.
  • Global: Removed system-ui from --font-sans; added oklch hex fallback comments; inherit font on form controls.

Verification

  • npm run build succeeded (Vite 6, 66 modules).
  • Manual curl to http://127.0.0.1:8000/query returns structured LLM JSON without llm_error.

Relevant for Interview

Demonstrates moving from mock-first UI to production-honest dashboard behavior without rewriting the design system.