Operation LongMemEval 80: overnight plan to chase SOTA

[Atharva-Kanherkar]

Operation LongMemEval 80

One-night plan to turn agentic-memory into a credible LongMemEval competitor and push toward the 80% band on LongMemEval_S cleaned.

Brutal Truth

One night is not enough to beat every mature memory product on every benchmark.

One night is enough to:

• make LongMemEval evaluation honest instead of distorted
• build a benchmark-specific adapter without throwing away the current architecture
• get a serious first-pass score on oracle
• create a path to the 75-80% band on LongMemEval_S cleaned

The right strategy is not "rewrite the memory system."

The right strategy is:

• keep the current semantic / episodic / procedural architecture
• add a LongMemEval adapter above it
• make timestamps, role structure, query planning, update handling, and abstention first-class in benchmark mode

Target

Primary target

• LongMemEval_S cleaned: 80% stretch target

Secondary targets

• LongMemEval oracle: 85%+
• LongMemEval_S cleaned: 70%+ minimum acceptable overnight outcome
• LongMemEval_M cleaned: do not optimize for this tonight

Success conditions

• honest ingest with preserved timestamps
• reproducible benchmark runner
• retrieval logs by question type
• answer generation with calibrated abstention
• publishable results, even if not yet SOTA

Strategy

Principle 1

Do not benchmark the generic SDK surface as-is. Build a benchmark adapter.

Principle 2

Do not rely on one retrieval path. Use separate indexes for:

• session summaries
• raw turns
• extracted user facts / preferences / updates

Principle 3

Do not answer directly from nearest-neighbor recall. Use:

• query-type classification
• candidate session narrowing
• turn-level evidence assembly
• answer synthesis with abstention

Principle 4

Do not mutate benchmark state during evaluation.

• no forgetting cycle
• no access-driven ranking changes
• no timestamp drift

Target Architecture

flowchart TD
    A[LongMemEval JSON] --> B[Benchmark Ingest Adapter]
    B --> C1[Turn Index<br/>episodic memories]
    B --> C2[Session Summary Index<br/>semantic memories]
    B --> C3[Fact / Preference / Update Index<br/>semantic memories]

    Q[Question] --> P[Query Planner]
    P --> P1[single-session]
    P --> P2[multi-session]
    P --> P3[temporal]
    P --> P4[knowledge-update]
    P --> P5[preference]
    P --> P6[abstention]

    P1 --> R[Retriever Orchestrator]
    P2 --> R
    P3 --> R
    P4 --> R
    P5 --> R
    P6 --> R

    C2 --> R
    C1 --> R
    C3 --> R

    R --> E[Evidence Pack Builder]
    E --> G[Reader / Answer Synthesizer]
    G --> J[Abstention Gate]
    J --> O[Hypothesis JSONL]
    O --> M[LongMemEval Evaluator]

Loading

Overnight Execution Plan

gantt
    title One-Night LongMemEval Sprint
    dateFormat  HH:mm
    axisFormat  %H:%M

    section Foundation
    Create benchmark mode and runner           :a1, 00:00, 00:45
    Preserve timestamps end-to-end            :a2, after a1, 01:00

    section Ingest
    Turn-level ingest adapter                 :b1, after a2, 01:15
    Session summary generation                :b2, after b1, 00:45
    Fact / preference extraction              :b3, after b1, 01:15

    section Retrieval
    Query planner by task type                :c1, after b2, 01:00
    Dual-stage retrieval                      :c2, after c1, 01:15
    Temporal and update-specific logic        :c3, after c2, 01:15

    section QA
    Evidence packing and answer prompt        :d1, after c2, 00:45
    Abstention calibration                    :d2, after d1, 00:45

    section Evaluation
    Oracle run and triage                     :e1, after d2, 00:45
    S cleaned run and triage                  :e2, after e1, 01:00
    Final tuning and report                   :e3, after e2, 00:45

Loading

Concrete Workstreams

1. Make benchmark ingest faithful

Tonight's minimum code changes:

• allow created_at to be passed through SDK / API / MCP / benchmark adapter
• encode each turn as its own episodic memory
• store role, session_date, and session_id in metadata
• keep turn_number
• keep a deterministic ingest order

Fastest implementation choice:

• do not redesign the stores
• use current EpisodicMemory plus metadata
• create a dedicated benchmark ingest path under benchmarks/longmemeval/

2. Build three indexes, not one

Turn index

Use for exact evidence retrieval.

• one memory per turn
• content format: role: content
• metadata: role, session_id, session_date, turn_number

Session summary index

Use for candidate session narrowing.

• one summary per session
• include session timestamp
• include compact list of important entities, events, preferences, and updates

Fact / preference / update index

Use for preference and knowledge-update questions.

• extract user facts and assistant-known facts from sessions
• attach validity date
• attach supersession / replacement info where possible

3. Add query planning

Classify each question into one of:

• single-session-user
• single-session-assistant
• single-session-preference
• temporal-reasoning
• knowledge-update
• multi-session
• abstention

Routing policy:

• single-session-*: search summaries first, then turns
• temporal-reasoning: infer anchor date/event, narrow sessions by time, then retrieve turns
• knowledge-update: search fact index, prefer latest valid fact, then verify supporting turn
• multi-session: retrieve top sessions, then merge turn evidence across sessions
• abstention: if evidence confidence is weak or contradictory, answer with abstention

4. Add temporal logic explicitly

The benchmark punishes generic vector search.

Need explicit handling for:

• first
• last
• before
• after
• latest
• current
• previous
• when did X happen

Implementation rule:

• use timestamp filtering before final reranking
• sort evidence chronologically when the query implies order
• never let access timestamps contaminate benchmark chronology

5. Add update handling explicitly

Current contradiction logic is useful but too manual for benchmark-grade updates.

Tonight's shortcut:

• extract candidate facts per session
• group by entity / attribute
• keep latest compatible fact
• mark older candidates as outdated in the fact index
• require turn evidence from the winning session before final answer

6. Add abstention as a real subsystem

LongMemEval includes abstention. Hallucinating kills score.

Simple overnight gate:

• if top evidence scores are weak, abstain
• if evidence conflicts and no winner is clear, abstain
• if retrieved evidence does not mention the asked entity / attribute, abstain

Deliverables By Dawn

Must ship

• benchmarks/longmemeval/run.py
• benchmarks/longmemeval/ingest.py
• benchmarks/longmemeval/query_planner.py
• benchmarks/longmemeval/retrieve.py
• benchmarks/longmemeval/answer.py
• benchmarks/longmemeval/prompts.py
• benchmarks/longmemeval/metrics.py
• benchmarks/results/ with oracle and S-cleaned outputs

Minimal runtime changes

• benchmark-safe timestamp ingest
• benchmark mode flag to disable retrieval-time mutations
• helper functions for filtering episodic memories by metadata and time

Repo Change Plan

flowchart LR
    A[Existing runtime] --> B[Small safe changes]
    B --> B1[created_at passthrough]
    B --> B2[benchmark mode disables state mutation]
    B --> B3[metadata helpers]

    A --> C[New benchmark package]
    C --> C1[ingest.py]
    C --> C2[query_planner.py]
    C --> C3[retrieve.py]
    C --> C4[answer.py]
    C --> C5[run.py]
    C --> C6[prompts.py]
    C --> C7[metrics.py]

    C --> D[Result artifacts]
    D --> D1[hypotheses.jsonl]
    D --> D2[evaluation summaries]
    D --> D3[error analysis]

Loading

Tuning Order

Do not tune everything at once.

1. Get oracle above 85%
2. Get single-session and preference stable
3. Fix temporal-reasoning
4. Fix knowledge-update
5. Calibrate abstention
6. Only then run a full S cleaned pass

Kill Criteria

If these are true after the first half of the night, stop chasing 80% and ship the strongest honest system:

• oracle stays below 70%
• session narrowing misses too many evidence sessions
• abstention is overfiring
• timestamps are still being stamped with wall-clock now

If that happens, optimize for:

• a publishable benchmark harness
• oracle quality
• clear roadmap to 80%

What Not To Do Tonight

• do not redesign the full memory model
• do not optimize multimodal retrieval
• do not chase LongMemEval_M
• do not lead with MCP
• do not spend hours on UI or docs polish

Outcome Ladder

Best case

• oracle: 90%+
• S cleaned: 78-82%
• credible claim that the architecture can compete

Good case

• oracle: 85%+
• S cleaned: 68-76%
• strong benchmark harness and a clear final-mile gap

Bad case

• oracle: <70%
• S cleaned: <60%
• still valuable, because the harness exposes exactly where the system fails

Final Instruction

Tonight is not about proving the generic memory product is perfect.

Tonight is about building the narrowest, sharpest, most benchmark-faithful system possible on top of the existing architecture, so the first serious LongMemEval result is strong enough to matter.