feat(pi-memory-mem0): observedAt + LoCoMo eval harness#77
Merged
Conversation
Mem0Provider.add() gains an optional observedAt (Date | string). mem0's extraction prompt documents an Observation Date used to ground relative time references, but never exposes a way to set it — it always falls back to the system clock, which mis-stamps facts when ingesting historical conversations. OSS mode now intercepts the extraction LLM call and rewrites the Observation/Current Date sections to observedAt (serialized via a mutex since the LLM wrapper is shared instance state); platform mode forwards it as the add() timestamp field. Omitting observedAt preserves current behavior for live turns. Adds tests/eval, a private workspace package that benchmarks pi-memory and pi-memory-mem0 against LoCoMo: forgiving dataset loader, token-F1 + LLM-judge scoring, retry/concurrency, and per-run isolated vector-store/snapshot paths so runs don't accumulate. On the 2-sample slice, wiring observedAt plus isolating the vector store moved LLM-judge recall from ~0.27 to ~0.44.
Replace the heuristic curated runner with an agent-style write loop: for each conversation session an LLM sees the turns plus the current memory snapshot and emits memory_add/replace/remove ops, applied to a real MemoryStore (2200/1375 char caps + threat scan enforced). Scoring uses the resulting snapshot under the same LLM-judge as the mem0 runner, so the two share a scale. Document the 2-sample results and, crucially, why the mem0 (~0.44) vs pi-memory (~0.19) judge gap is not a leaderboard: LoCoMo rewards exhaustive detail recall, which pi-memory deliberately discards under its char budget. Also note the per-run vector-store isolation and observedAt fixes behind mem0's jump from ~0.27, judge variance, and the reasoning-model max_tokens requirement.
workflow_dispatch-only job that runs the LoCoMo eval on the self-hosted runner. Inputs pick the runner (mem0 / curated / both), sample count, top-k, and the extraction/judge models. Reuses PI_INTEGRATION_BASE_URL/API_KEY to write a temp models.json (provider deepseek-integration), fetches LoCoMo, runs the eval + LLM-judge, writes a summary table to the job summary, and uploads results/*.json as an artifact. Uses prebuild-better-sqlite3 for mem0's OSS SQLite vector store.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a memory-eval harness (
tests/eval, private workspace package) that benchmarkspi-memoryandpi-memory-mem0against LoCoMo, and a product change topi-memory-mem0that fell out of building it.pi-memory-mem0:
observedAtMem0Provider.add()gains an optionalobservedAt(Date | string). mem0's extraction prompt documents an "Observation Date" used to ground relative time references ("yesterday", "last week"), but never exposes a way to set it — it always falls back to the system clock, which mis-stamps facts when ingesting historical conversations.observedAt(serialized via a mutex, since the LLM wrapper is shared instance state).add()timestampfield.observedAtpreserves current behavior for live turns.Exported
formatObservedAt/rewriteObservationDatewith unit tests (+6).tests/eval
memorystore is SQLite-backed at~/.mem0/vector_store.dband silently accumulates across runs).memory_add/replace/removeops applied to a realMemoryStore(char caps + threat scan enforced).recallJudge.models.json.Results (LoCoMo 2-sample, flash extract + pro judge)
Not a leaderboard — LoCoMo rewards exhaustive detail recall, which pi-memory deliberately discards under its char budget. The README explains the mismatch. mem0's score rose from ~0.27 after the vector-store isolation and
observedAtfixes.Notes
private, not published.