Skip to content

graphiti-ingestion-status.md

Latest commit

 

History

History
138 lines (99 loc) · 3.76 KB

graphiti-ingestion-status.md

File metadata and controls

138 lines (99 loc) · 3.76 KB
Raw

Graphiti Ingestion Status

Date: 2026-03-21 (Saturday morning autonomous reflection) Status: ✅ Parallel ingestion implemented, ⚠️ blocked on OpenAI quota

What Was Done

Parallel Ingestion Implementation (Issue #153)

Implemented parallel message processing in scripts/paced_ingestion.py:

  • New --parallelism parameter: Default 12 (matches NUC capacity)
  • Chunked async processing: Uses asyncio.gather() to process messages concurrently
  • All error handling preserved: Per-message tracking, halt logic, logging intact
  • Test validated: Processed 9 messages in 3 parallel chunks, all error handling worked correctly

Performance Impact:

  • Before: Sequential processing, ~78s per message
    • 1,334 messages × 78s = 40+ hours
  • After: Parallel processing, 12 messages at a time
    • 1,334 messages / 12 × 78s = ~8,658s = 2.4 hours
  • Speedup: ~17x faster

Commit: 60d955c - "feat(graphiti): add parallel ingestion to fix 40-hour backlog"

Current Blocker: OpenAI Embedding Quota

Test run result: 9/9 messages failed with:

Error code: 429 - insufficient_quota
'You exceeded your current quota, please check your plan and billing details.'

Configuration (pps/docker/.env):

  • Provider: openai
  • Model: text-embedding-3-small
  • Dimensions: 1024
  • Key: Jeff's PPS project key (rotated 2026-02-23, $10 credit + auto-recharge capped)

The credits ran out.

Decision Needed

Two paths forward:

Option 1: Add OpenAI Credits (Keep Hybrid Mode)

Pros:

  • Quick unblock (~5 minutes)
  • Keep existing graph (24,612 messages already ingested)
  • Hybrid mode working well (NUC LLM + OpenAI embeddings)

Cons:

  • Ongoing cost (minimal for text-embedding-3-small at 1024-dim)
  • Dependency on external service

Action: Add credits to OpenAI account, run ingestion

Option 2: Switch to Local Embeddings (Fully Local)

Pros:

  • Zero ongoing cost
  • Full local control
  • NUC has capacity (nomic-embed-text-v1.5 validated)

Cons:

  • Requires nuking existing graph (vector spaces incompatible)
  • Need to re-ingest all ~26,600 messages (24,612 + 1,975 pending + 271 failed)
  • Time cost: ~26,600 messages / 12 × 78s = ~4.7 hours
  • Risk: losing existing graph structure (19,000+ messages of relationships)

Action:

  1. Backup Neo4j database
  2. Update .env to local embeddings (nomic-embed-text-v1.5, 768-dim)
  3. Wipe Neo4j graph
  4. Re-ingest all messages from scratch

Config change:

# pps/docker/.env
GRAPHITI_EMBEDDING_PROVIDER=local
GRAPHITI_EMBEDDING_MODEL=text-embedding-nomic-embed-text-v1.5
GRAPHITI_EMBEDDING_DIM=768

Recommendation

Option 1 (add credits) is recommended:

  1. Faster unblock (minutes vs hours)
  2. Preserves existing graph quality
  3. Cost is minimal for embeddings
  4. Can always switch to local later if needed (will require graph wipe then too)

Next Steps (After Decision)

Once credits are added OR local embeddings configured:

  1. Small validation run:

    python scripts/paced_ingestion.py --parallelism 3 --batch-size 9 --max-batches 1

  2. Check results:

    • Verify messages ingested successfully
    • Check Neo4j for new entities/relationships
    • Review logs for errors

  3. Full backlog ingestion:

    python scripts/paced_ingestion.py --parallelism 12 --batch-size 50 --pause 10

  4. Monitor:

    • Watch scripts/ingestion.log
    • Check progress: "Progress: X ingested, Y failed, Z pending"
    • Expected duration: ~2.4 hours

Files Modified

  • scripts/paced_ingestion.py - Added parallel processing
  • This status doc

Issues

  • #153 - Parallel ingestion (implementation complete, testing blocked)
  • Graph backlog: 1,975 messages pending (was 1,334 when I started, grew during testing)