A no-context baseline scores 49.2% on PersonaMem-128k (v1), including 80% on `track_full_preference_evolution`

[Paul-Kyle]

Summary

The paper's data-curation section states that questions answerable without context are excluded:

we exclude all questions that the model can answer correctly without seeing any contextual information

Running a no-context baseline on the released v1 128k data — question and answer options only, no conversation history, no retrieval, no memory — a frontier model scores 49.2% (232/472) against a 25% four-option chance floor.

The leak is uneven, and it is largest on the preference-update question types.

Method

We kept the prompt and scoring deliberately identical to the paper so those
parts of the evaluation are comparable:

• Prompt: the paper's own instruction, verbatim (Find the most appropriate model response…).
• Scoring: the paper's scoring rule, verbatim.
• Input: user_question_or_message + all_options only. No conversation, no shared context, no retrieval, no memory.
• Dataset: questions_128k.csv from PersonaMem-v1 revision 73dfd752, downloaded 2026-08-05.
• n: 472 questions from the 128k tier. One response was unparseable (0.2%) and is counted as incorrect.

Measurement limitations, stated up front

• Model identification. The run used a GPT-5.6-class frontier reasoning model reached through a consumer subscription interface, not a pinned API snapshot. We therefore cannot certify the exact model revision behind the number.
• No temperature control. That interface exposes no sampling controls, so this is not a temp=0 measurement.
• Neither limitation plausibly explains a 24-point gap above the chance floor, but both mean the precise figure should be treated as approximate. We would happily re-run against a keyed API if that would make the report more useful to you.
• We also ran the same no-context arm on an open-weights ~26B model over a 55-question subset and got 40.0%. Paired on those same 55 question IDs the two models differ by 40.0% vs 45.5% with 9 vs 6 discordant pairs — we detected no difference on this small matched subset (exact two-sided paired p = 0.607). That is a failure to detect a difference, not evidence that the models behave equivalently; the subset is small.

Results by question type

question type	no-context accuracy	n
track_full_preference_evolution	80.0%	55
revisit_reasons_behind_preference_updates	69.8%	43
provide_preference_aligned_recommendations	47.5%	59
acknowledge_latest_user_preferences	45.1%	142
recall_user_shared_facts	42.9%	35
suggest_new_ideas	42.4%	99
generalize_to_new_scenarios	23.1%	39

The two highest are the preference-update axis. At 80% and 70% answerable with no history, there is little headroom on those types for a memory system to demonstrate anything.

An associated pattern: question length

No-context accuracy is positively associated with question length: per-type Pearson r = +0.88 (7 types), per-question point-biserial r = +0.30 (n=472).

question type	no-context acc	mean chars
track_full_preference_evolution	80.0%	709
revisit_reasons_behind_preference_updates	69.8%	453
provide_preference_aligned_recommendations	47.5%	165
acknowledge_latest_user_preferences	45.1%	74

The relationship is not a smooth gradient — by length quartile, accuracy runs 43.2% / 49.2% / 34.7% / 69.5%, so it is carried by the longest questions rather than rising steadily.

We have not tested length against other candidate predictors, so this is an observed association, not a claim that length is the dominant cause. A plausible mechanism: the longest in-situ user messages restate the user's own history inside the question text, so the question supplies context the conversation was meant to supply.

Suggested remedies

1. Re-run the exclusion filter with a stronger model. If it was applied with a weaker one, questions a frontier model finds easy would have survived it.
2. Publish the filter's model and threshold in the README or dataset card, so users can reason about what "answerable without context" meant operationally.
3. Consider a length cap or rewrite pass on the longest in-situ message types.
4. Ship a no-context baseline as a reported control alongside the leaderboard, so readers can separate memory capability from question-carried context.

Scope and provenance

We are evaluating an external memory system against PersonaMem and hit this while sanity-checking our own results, which came in well above the published frontier leaderboard. We have not decomposed how much of that gap this accounts for — the two are plausibly related but we cannot attribute it quantitatively, so we are reporting only the no-context measurement itself.

Happy to share the 472 per-question outputs (question ID, type, chosen option, gold, correctness) so you can verify or reproduce directly. We can also re-run against a keyed API using the same pinned dataset revision.

We have not checked whether v2 shows the same property, and note that it explicitly targets implicitly-revealed preferences, which may address this mechanism by construction.