Skip to content

README.md

Latest commit

 

History

History
142 lines (109 loc) · 5.39 KB

README.md

File metadata and controls

142 lines (109 loc) · 5.39 KB
Raw

Benchmarks

Mnemosyne includes benchmark harnesses for retrieval quality. The current published memory-system result uses LongMemEval, a benchmark where each question has its own candidate set of conversation sessions and one or more answer session IDs.

Headline Results

These runs use local inference only: no cloud API and no LLM reranker.

Benchmark Document Mode Model Config Fusion Recall@5 Recall@10 MRR@10 nDCG@10
LongMemEval message jina-v5-nano vector-bm25, weight 0.10 0.9880 0.9980 0.9586 0.9575
LongMemEval session jina-v5-nano vector-bm25, weight 0.10 0.9920 0.9940 0.9491 0.9486

The message run is the more representative Mnemosyne result: each user/assistant message is stored as a short document, and retrieved documents are scored by whether they map back to the correct LongMemEval answer session. The session run is included because it is the closest shape to MemPalace's raw LongMemEval setup: one document per session.

Full generated result files:

Reproduce LongMemEval

Build Mnemosyne first:

task build

Run a small mechanics check:

python3 benchmarks/longmemeval/run.py \
  --config ./configs/jina-v5-text-nano-retrieval.yaml \
  --run-label jina-v5-nano \
  --doc-mode message \
  --max-queries 2 \
  --max-sessions 8 \
  --no-rerank

Run the published short-document result:

python3 benchmarks/longmemeval/run.py \
  --config ./configs/jina-v5-text-nano-retrieval.yaml \
  --run-label jina-v5-nano \
  --doc-mode message \
  --no-rerank \
  --limit 50 \
  --rerank-candidates 50

Run the session-document comparison:

python3 benchmarks/longmemeval/run.py \
  --config ./configs/jina-v5-text-nano-retrieval.yaml \
  --run-label jina-v5-nano \
  --doc-mode session \
  --no-rerank \
  --limit 50 \
  --rerank-candidates 50

The harness downloads longmemeval_s_cleaned.json into benchmarks/data/longmemeval/ unless the file already exists or --skip-download is used. Scratch databases are written under benchmarks/work/longmemeval/. Result summaries are written to benchmarks/results/.

The full message-mode run imports 246,738 short documents across 500 isolated per-question databases and took about 7.1 hours on the original local run. The session-mode run imports 23,796 documents and took about 1.3 hours.

Use --reuse-db only when the per-question databases for the same document mode, model config, and collection already exist.

Metrics

For LongMemEval, each retrieved document is mapped back to its longmemeval_session_id metadata value before scoring.

  • recall_any@5 / recall_any@10: at least one correct answer session appears in the top 5 or top 10.
  • recall_all@5 / recall_all@10: all labelled answer sessions appear in the top 5 or top 10. This is stricter for multi-session questions.
  • MRR@10: reciprocal rank of the first correct answer session in the top 10. Higher means the first useful result appears earlier.
  • nDCG@10: ranking quality over the top 10, giving more credit when relevant sessions are near the top.

The generated Markdown files also include first-relevant-rank buckets, per-question-type scores, and misses at 5.

Comparison Notes

These numbers are retrieval metrics, not end-to-end question-answering accuracy. They measure whether Mnemosyne retrieves the right memory session; they do not measure whether an LLM can answer correctly after retrieval.

For a rough local-only comparison, MemPalace reports LongMemEval raw ChromaDB at 96.6% R@5 / 98.2% R@10 / 0.889 nDCG@10 and hybrid v2 at 98.4% R@5 / 99.0% R@10 / 0.934 nDCG@10. Mnemosyne's message-mode result is 98.8% Recall@5, 99.8% Recall@10, and 0.9575 nDCG@10 without an LLM reranker.

The comparison is not perfectly apples-to-apples:

  • Mnemosyne reports recall_any@K, plus the stricter recall_all@K for questions with multiple answer sessions.
  • MemPalace also reports higher LLM-reranked numbers; those add a reader model after retrieval, while the Mnemosyne results above do not.
  • MemPalace's own benchmark documentation flags its 100% LongMemEval result as partly tuned on known failures, and separately publishes a clean held-out no-LLM hybrid result of 98.4% R@5.

BEIR SciFact

The BEIR harness is useful for standard document-retrieval checks on SciFact:

task bench:scifact-smoke
task bench:scifact

Useful flags:

task bench:scifact -- --no-rerank
task bench:scifact -- --reuse-db
task bench:scifact -- --fts-only --no-rerank
task bench:scifact -- --vector-only --no-rerank
task bench:scifact -- --fusion vector-bm25 --bm25-weight 0.10 --rerank-candidates 300 --no-rerank