troubleshooting.md

Troubleshooting

Common issues and solutions.

Quick Diagnostics

# Check if server is running
curl http://localhost:7474/health

# Check logs
docker logs nornicdb
# or
journalctl -u nornicdb -n 100

# Check resources
docker stats nornicdb
# or
htop

Connection Issues

Cannot Connect to Server

Symptoms:

• Connection refused
• Timeout

Solutions:

1. Check if server is running:

   curl http://localhost:7474/health

2. Check bind address:

   # Docker requires 0.0.0.0
   NORNICDB_ADDRESS=0.0.0.0

3. Check ports:

   netstat -tlnp | grep 7474

4. Check firewall:

   sudo ufw status
   sudo firewall-cmd --list-ports

Connection Reset

Symptoms:

• Intermittent disconnects
• "Connection reset by peer"

Solutions:

1. Check rate limiting:

   # View rate limit hits
   curl http://localhost:7474/metrics | grep rate_limit

2. Increase limits:

   rate_limiting:
     per_minute: 200
     per_hour: 6000

Authentication Issues

401 Unauthorized

Symptoms:

• All requests return 401

Solutions:

1. Check token:

   # Get new token
   curl -X POST http://localhost:7474/auth/token \
     -d "grant_type=password&username=admin&password=admin"

2. Check auth is disabled (dev only):

   NORNICDB_NO_AUTH=true

3. Check JWT secret:

   # Must be at least 32 characters
   NORNICDB_JWT_SECRET="your-32-character-secret-key-here"

403 Forbidden

Symptoms:

• Authenticated but access denied

Solutions:

1. Check user role:

   # User needs appropriate permissions
   nornicdb user update alice --role editor

2. Check endpoint permissions:

   • /status requires read permission
   • /metrics requires read permission
   • Admin endpoints require admin permission

Performance Issues

Slow Queries

Symptoms:

• High latency
• Timeouts

Solutions:

1. Enable query caching:

   nornicdb serve --query-cache-size=5000

2. Check query complexity:

   // Bad: Unbounded traversal
   MATCH (n)-[*]->(m) RETURN n, m
   
   // Good: Limited depth
   MATCH (n)-[*1..3]->(m) RETURN n, m LIMIT 100

3. Add indexes:

   CREATE INDEX FOR (n:Person) ON (n.email)

4. Enable parallel execution:

   nornicdb serve --parallel=true

Maximum query/content size (MCP + embeddings)

If you are using the MCP tools (store, discover, etc.):

• HTTP request size: MCP limits request bodies via MaxRequestSize (default 10MB).
  • Applies to /mcp, /mcp/tools/call, and /mcp/initialize.
• Embedding input limits: the effective max “query size” for vector search is bounded by the embedding model context.
  • For local GGUF embeddings, long queries are chunked into embedding-safe pieces and searched across all chunks (results are fused), so vector search can still work with paragraph-sized queries.
• Stored content: large content is chunked for embeddings (default 512 tokens per chunk with overlap), but very large payloads will increase background embedding work.

High Memory Usage

Symptoms:

• OOM errors (exit code 137 in Docker)
• Container restarts repeatedly
• Slow performance

Solutions:

1. Enable Low Memory Mode (recommended):

   # Reduces BadgerDB RAM from ~1GB to ~50MB
   nornicdb serve --low-memory
   
   # Or via environment variable
   NORNICDB_LOW_MEMORY=true nornicdb serve

   See Low Memory Mode Guide for details.

2. Increase GC frequency:

   nornicdb serve --gc-percent=50

3. Enable object pooling:

   nornicdb serve --pool-enabled=true

4. Docker: Increase memory limit:

   deploy:
     resources:
       limits:
         memory: 4G

5. Docker: Check for WAL bloat:

   # Large WAL files can cause OOM on startup
   du -sh /path/to/data/wal/
   
   # If >1GB, consider trimming it.
   #
   # NOTE: Deleting the WAL can lose the *very latest* mutations if the process
   # crashed after WAL append but before the change was applied to the main store.
   # Prefer auto-recovery (snapshot + WAL) when possible.
   rm /path/to/data/wal/wal.log

Data Integrity / Recovery

Server won't start until data is deleted (corruption after crash)

Typical trigger:

• Hard kills (OOM / docker kill -9), GPU/CGO segfaults, host power loss, or storage hiccups.

What to do:

1. Do not delete your data directory. Instead, back it up first.

2. Use snapshot + WAL recovery (Neo4j-style “tx log recovery”)

NornicDB maintains:

• Snapshots in <dataDir>/snapshots/ (e.g., snapshot-YYYYMMDD-HHMMSS.json)
• WAL in <dataDir>/wal/wal.log

Auto-recovery (recommended)

Auto-recovery is enabled by default. You can:

• Disable: NORNICDB_AUTO_RECOVER_ON_CORRUPTION=false
• Force an attempt (even if the open error message doesn’t match the corruption heuristics): NORNICDB_AUTO_RECOVER_ON_CORRUPTION=true

NORNICDB_AUTO_RECOVER_ON_CORRUPTION=true

On startup, NornicDB will:

• Repair the WAL tail if the previous process crashed mid-write
• Load the latest snapshot (if present)
• Replay WAL entries after that snapshot (or replay WAL-only if no snapshots exist yet)
• Rename your original directory to <dataDir>.corrupted-<timestamp> (for forensics)
• Rebuild a fresh store and restore recovered nodes/edges

3. If recovery can’t run

• Ensure the container has write access to the volume
• Ensure at least one recovery artifact exists:
  • snapshots in <dataDir>/snapshots/, and/or
  • WAL in <dataDir>/wal/ (wal.log or sealed segments/seg-*.wal)
• Avoid running the DB data directory on unstable/union filesystem mounts (prefer a dedicated disk path)

High CPU Usage

Symptoms:

• CPU at 100%
• Slow responses

Solutions:

1. Limit parallel workers:

   nornicdb serve --parallel-workers=2

2. Check for expensive queries:

   # Enable query logging
   nornicdb serve --log-queries

3. Check embedding queue:

   curl http://localhost:7474/status | jq .embeddings

Data Issues

Data Not Persisting

Symptoms:

• Data lost on restart

Solutions:

1. Check volume mount:

   docker inspect nornicdb | grep Mounts

2. Check data directory:

   ls -la /data

3. Check disk space:

   df -h

Corrupted Data

Symptoms:

• Read errors
• Inconsistent results

Solutions:

1. Verify data:

   nornicdb verify --check-all

2. Restore from backup:

   nornicdb restore --input backup.tar.gz

3. Rebuild indexes:

   nornicdb admin rebuild-indexes

Embedding Issues

Embeddings Not Generating

Symptoms:

• Nodes without embeddings
• Search not working

Solutions:

1. Check embedding service:

   curl http://localhost:11434/api/embed \
     -d '{"model":"mxbai-embed-large","input":"test"}'

2. Check configuration:

   NORNICDB_EMBEDDING_URL=http://ollama:11434
   NORNICDB_EMBEDDING_MODEL=mxbai-embed-large

3. Check pending queue:

   curl http://localhost:7474/status | jq .embeddings.pending

4. Trigger regeneration:

   curl -X POST http://localhost:7474/nornicdb/embed/trigger?regenerate=true

Docker Issues

Container Won't Start

Solutions:

1. Check logs:

   docker logs nornicdb

2. Check image:

   docker pull timothyswt/nornicdb-arm64-metal:latest

3. Check resources:

   docker system df

Permission Denied

Solutions:

1. Fix volume permissions:

   docker run --rm -v nornicdb-data:/data busybox chown -R 1000:1000 /data

2. Run as root (not recommended):

   security_opt:
     - no-new-privileges:false

Getting Help

Collect Diagnostics

# System info
uname -a
docker version
go version

# NornicDB info
curl http://localhost:7474/status

# Logs
docker logs nornicdb > nornicdb.log 2>&1

Log Locations

Deployment	Location
Docker	docker logs nornicdb
Systemd	journalctl -u nornicdb
Binary	./nornicdb.log

See Also

• Monitoring - Health monitoring
• Deployment - Deployment guide
• Scaling - Performance tuning