Option length predicts the correct answer in PersonaMem-v2: gold is among the shortest options in all 52 sampled `who=others` rows and among the longest in 46.7% overall

[Paul-Kyle]

Summary

Following our earlier no-context report on v1, we ran the same gate on
PersonaMem-v2. The good news first: v2 is a real improvement on the property we
reported for v1 — a frontier model with no conversation history scores 31.1%
[27.1, 35.2] here against the 25% four-option floor, versus 49.2% on v1.

But while checking that, we found a different leak: the length of the answer
options predicts the correct answer. Even under a tie-conservative count — a
row counts only when the gold is the unique longest option — the signal scores
44.7% overall, above the frontier model's 31.1%. On one slice it is nearly
deterministic.

• The gold is among the longest options in 46.7% overall, and uniquely
  longest in 44.7% (vs 25% chance).
• On who=others, the gold is among the shortest options in 52 of 52
  sampled rows, and uniquely shortest in 51 of 52.

These are properties of the released data, not of any model, so they can be
verified directly (snippet below).

Method

Stratified sample of n=501 across all 7 pref_type values from the released
v2 text benchmark at revision
b7b42b7,
question and options only — no history, no retrieval. Prompt and scoring
replicated from the released inference.py: the authors'
create_mcq_options() wording and structure, Final Answer: [Letter] strict
parsing (a bare letter inside reasoning prose does not count; 0.8% unparsed,
counted incorrect). Two documented deviations: the MCQ block is concatenated
into the user message rather than sent as a trailing system message (there
is no history to separate them in a no-context run), and the option shuffle
uses an md5-derived seed because Python's hash() is salted per process and
not reproducible. The runner is rooted at an empty directory so no local
context can leak into answers.

The length signal, by slice

The direction of the association inverts by slice (all numbers on the n=501
sample). The two length columns count whether gold is among the extrema;
ties are exposed separately by the verification snippet below.

slice	frontier, no context	gold among longest	gold among shortest	n
who=others	9.6%	0.0%	100.0%	52
anti_stereotypical_pref	36.0%	68.6%	8.1%	86
stereotypical_pref	34.0%	67.9%	5.7%	53
neutral_preferences	33.7%	66.3%	9.3%	86
health_and_medical_conditions	33.3%	61.4%	10.5%	57
sensitive_info	66.7%	56.9%	0.0%	51
ask_to_forget	14.3%	3.8%	56.2%	105
therapy_background	15.9%	22.2%	50.8%	63
ALL	31.1%	46.7%	23.0%	501

A mechanism consistent with inspection: where the correct response declines
to use a preference — a forgetting request, someone else's preference — the
correct answer is a short acknowledgement while the distractors are longer
elaborations that wrongly apply the information. Where the correct response
uses a preference, it is the richest option and the distractors are thinner.
We have not tested this against other candidate predictors; we are reporting
the association and the exploitable rules, not a causal decomposition.

Verify without a model

On the pinned released data, per row: parse incorrect_answers, compare
character lengths against correct_answer, group by who / pref_type, and
report both extrema membership and the tie-conservative unique-extrema rates:

import ast, collections, csv, io, sys, urllib.request

csv.field_size_limit(sys.maxsize)
REV = "b7b42b78917157afed063527a1c959e98f6109f2"
URL = f"https://huggingface.co/datasets/bowen-upenn/PersonaMem-v2/resolve/{REV}/benchmark/text/benchmark.csv"
tally = collections.defaultdict(lambda: [0, 0, 0, 0, 0])
rows = csv.DictReader(io.TextIOWrapper(urllib.request.urlopen(URL), encoding="utf-8"))
for r in rows:
    lens = [len(r["correct_answer"])] + [len(x) for x in ast.literal_eval(r["incorrect_answers"])]
    t = tally[(r["who"], r["pref_type"])]
    is_long = lens[0] == max(lens)
    is_short = lens[0] == min(lens)
    t[0] += is_long
    t[1] += is_long and lens.count(max(lens)) == 1
    t[2] += is_short
    t[3] += is_short and lens.count(min(lens)) == 1
    t[4] += 1
for k, (long, unique_long, short, unique_short, n) in sorted(tally.items()):
    print(k, f"gold-among-longest {long/n:.1%}",
          f"unique-longest {unique_long/n:.1%}",
          f"gold-among-shortest {short/n:.1%}",
          f"unique-shortest {unique_short/n:.1%}", f"n={n}")

Our 52/52 is on the stratified sample; this runs the check on the full set.

Why we think the supersession/forgetting axis is still valuable

This is not a "v2 is unusable" report. updated=True is the same axis as
ask_to_forget in the released text benchmark: all 1,047 updated=True rows
fall within its 1,048 ask_to_forget rows. On this supersession/forgetting
axis, a frontier model with no context scores 14.3% — well below chance,
because it is drawn to whichever option sounds most personalized and those are
the distractors. The slice therefore retains useful headroom for paired
memory-conditioned comparisons, with a known model-free baseline (56.2% gold
among shortest; 55.2% uniquely shortest on the sample) rather than an invisible
confound. We have been using ask_to_forget for paired A/B comparisons of
forgetting mechanisms — reporting only paired deltas between arms, never
absolute scores against the 25% floor — and it discriminates well for that
purpose.

Suggested remedies

1. Report the longest / shortest controls, including tie-conservative unique
   extrema, alongside every leaderboard number — they cost nothing and expose
   the operative no-model baseline.
2. Length-balance the options, or publish a length-matched subset.
3. Consider dropping or rebuilding who=others if the full-set check
   confirms the sampled near-determinism.
4. Document the intended no-model baseline per slice — 25% is not the only
   relevant comparator anywhere we measured.

Scope and provenance

Same posture as our v1 report: we hit this while gating the benchmark before
using it to evaluate an external memory system, and we are reporting the
measurement rather than an attribution. Happy to share the 501 per-question
no-context outputs (question ID, type, chosen option, gold, correctness), and
the frontier caveat from the v1 report applies to the 31.1% number only — the
length findings are model-free and reproducible from the dataset alone.