Measured effect on short vs long tasks, plus two silent failure modes from extension conflicts

[tim-korso]

Environment

• Pi 0.84.2 (pinned) and Pi 0.85.1 (current global), Node v26.7.0, npm 11.19.0, macOS, Apple Silicon
• SoL-Pi main @ d7ecfc089944f0d04b80122a0a9a6ca0d786f3d0, version 0.1.0
• Model backend: deepseek/deepseek-v4-flash via openai-completions (not a frontier model, and not xhigh)
• Pi installed in an isolated prefix; agent dir isolated via PI_CODING_AGENT_DIR. Upstream Pi was not patched.
• Config: actionFusion: true, observationPack: true, evidencePreservingReducer: false, onlineContextCompact: false

What I verified as working

Thanks for the held-out methodology and for shipping a no-Pi-patches extension. The following all passed on the pinned release:

• npm ci --ignore-scripts — 289 packages
• npm run typecheck — clean
• npm test — 18 files / 139 tests passed
• npx vitest run tests/all-mechanisms.test.ts — 4/4
• node scripts/check-pi-compat.mjs — exit 0
• scripts/check-sol-pi-config.mjs — correctly rejects unknown keys, version != 1, non-boolean flags, and partial config under --require-all-enabled (all exit 1)
• Pi startup with --offline --approve, no extension load error

Both local mechanisms produce real, auditable effect:

Action Fusion — edit with then_run executes and returns one fused observation:

Successfully replaced 2 block(s) in calc.js.[then_run:succeeded]
PASS add(2,3): got 5, expected 5
...
ALL TESTS PASSED

Turns dropped 4 -> 3 versus stock Pi on the same task.

ObservationPack — archive and ledger are real. ledger.jsonl:

{"event":"placeholder","sendNumber":3,"originalBytes":11499,"originalTokens":2875,"placeholderTokens":336,"removedTokens":2539,"tool":"bash"}

On a longer trajectory a single replacement removed 12,472 tokens (12,822 -> 350).

Numbers I could not reproduce, and what I think the gap is

I measured the same task under stock Pi and SoL-Pi, parsing per-turn usage from --mode json.

Task shape	Turns	Token delta
Small file, seconds	4-6	-8.1% over 3 pairs (one pair was +0.1%)
5 modules x 12 KB, minutes	5	-21.1% (57,967 -> 45,772)

README claims 45-49% fewer tokens than Pi. I believe the gap is mostly scope, not a defect:

1. Trajectory length. README states the mechanisms are measured over multi-hour trajectories, and explicitly notes shorter benchmarks do not match the research question. My tasks run in seconds to minutes. The delta scales with length in my own data (-8% -> -21%), which is consistent with your thesis.
2. Config scope. I enabled 2 of 4 mechanisms. The README figure is the assembled harness with all four.
3. Reasoning effort and backend. All your comparisons ran at xhigh on frontier backends. I ran a non-frontier model at a low thinking level, which likely generates less repeated context to compress.

So this is not a contradiction, but I think the README would set expectations better if the 45-49% were labeled with its preconditions (all four mechanisms, multi-hour trajectories, xhigh).

Actionable findings

1. then_run is not used autonomously — the mechanism is model-opt-in, not harness-forced.

This is the largest practical gap I hit. With SoL-Pi loaded and actionFusion: true, the model still issued a separate edit followed by a separate bash on the first attempt:

--- stock Pi ---     bash, edit, bash
--- SoL-Pi ---       bash, edit, bash      <- then_run absent, zero savings

Action Fusion only fired when I explicitly instructed the model to use the then_run parameter in the same call. Net token change on that run was +14% (8,508 -> 9,914): the longer edit/write schemas cost more, and the saved round-trip never materialized.

Since base-Pi rollouts are the stated motivation, the tool description probably needs to be much more directive (or the fusion offered as a system-prompt hint), otherwise users may see a small net loss on short tasks rather than a gain.

2. Issue #20 (then_run silently no-ops on Pi 0.85.1) is not reproducible on my machine.

I tested Pi 0.85.1 with SoL-Pi loaded, including the exact repro from the report (then_run: {"command": "cat <same-file>", "timeout": 15}):

emitted  then_run = {"command":"cat /private/tmp/solpi-bench/calc.js","timeout":15}
result   Successfully replaced 1 block(s)...[then_run:succeeded]
         export function mean(values) { ... return sum / (values.length); }

The command ran and the output was appended, identical to 0.84.2. I could not reproduce the silent failure either with or without the timeout field. This may be environment-specific (the reporter used npx and a SoL-Pi-only profile); it may be worth asking for their exact invocation, or closing it if it was a local artifact. I am flagging it rather than claiming it is fixed.

3. --no-session produces a hard extension failure rather than degrading.

Extension error (.../src/sol-pi/index.ts): SoL-Pi requires a persistent Pi session directory

This comes from runtime-paths.ts:11. --no-session / --session-dir omission is a legitimate Pi invocation, and a thrown error aborts the extension load. It would be friendlier to disable the archiving mechanisms with one warning and keep the rest of the harness working.

4. There is no per-mechanism ablation, so the 45-49% cannot be attributed.

The README gives one combined figure for four mechanisms. The swarm section already acknowledges this ("It does not by itself isolate which SoL-Pi mechanism caused the observed gain"). Since contributors are invited to improve individual mechanisms, a per-mechanism or leave-one-out table on a held-out subset would make the claim actionable. My own measurement suggests Action Fusion and ObservationPack behave very differently by task length.

5. peerDependencies accepts any Pi version, which is what #11 and #15 report.

All four Pi peer deps are "*". Combined with the finding that the practical compatibility boundary is a moving target (see #20), a floor such as >=0.84.2 would fail loudly at install time instead of at runtime.

Both mechanisms can be silently disabled by other extensions

Worth adding to the list above, because I hit it on a real setup and it took payload-level
instrumentation to diagnose. Neither mechanism warns when it loses its slot.

a. A competing editor tool makes Action Fusion a no-op.

A common third-party extension ships its own editor named str_replace_editor and hides the
built-in edit/write from the model. With SoL-Pi installed and actionFusion: true, the wire
tool list still contains edit, registration succeeds, and the then_run schema is present — but
the model picks str_replace_editor instead, and then_run is silently ignored:

tool_call: str_replace_editor  hasThenRun=True     <- argument accepted by nobody
tool_call: bash                hasThenRun=False
result:    Successfully replaced 1 block(s) in calc.js.      <- no [then_run:*] marker

So on such a setup Action Fusion costs schema tokens and delivers nothing, with no signal that it
is inert. This is a different case from #20: there the argument is dropped between model and
execution, here the model never chooses the tool SoL-Pi replaced.

b. A third-party setActiveTools() snapshot removes obs_recall.

Another extension in my config calls pi.setActiveTools(snapshot) using a tool list captured at
its own load time. If that extension loads before SoL-Pi, the snapshot predates the
obs_recall registration and the tool is dropped from every provider request for the entire
session. Only packages order changes the outcome:

SoL-Pi first:  ["read","bash","edit","write","obs_recall","str_replace_editor"]
other first:   ["read","bash","edit","write","str_replace_editor"]      <- obs_recall gone

Reordering fixed it. I have filed the root cause against that extension separately, but from
SoL-Pi's side the lesson is that registerTool succeeding does not imply the tool reaches the
model. Since the project's own methodology already includes "Prove mechanism activation" (MV15)
and "Evaluate a mechanism at its exact first firing point" (M8), a lightweight self-check that
reports when a replacement tool is not in the wire payload (or when then_run never fires in a
session) would surface both cases immediately instead of after a benchmark anomaly.

Reproducibility caveat

One trial per condition in my numbers as well; I observed -17.1%, +0.1% and -5.7% across three identical pairs, so short-task variance is large and single-run comparisons in this regime are not reliable. I am reporting the aggregate rather than any single run.

Happy to re-run any of this against a frontier backend at xhigh if that would make the comparison fairer, and to retest #20 on a second machine if useful.

[xqicxx]

Thanks for the measurements — finding 1 sent me looking, and I can confirm your "silent failure mode from extension
conflicts" concretely, with a named culprit.

Pi 0.85.1, SoL-Pi 0.1.0 @ d7ecfc0, actionFusion: true, deepseek-v4.1-flash @ max thinking, isolated
PI_CODING_AGENT_DIR. Same prompt, then_run never mentioned:

• SoL-Pi only: 5 / 5 edit calls carried then_run: {"command": "node calc.js", "timeout": 30}, all five returned
  [then_run:succeeded] with the command output appended.
• SoL-Pi + the 15 extensions I normally run: 36 tool calls, every one fabric_exec; then_run appears zero times and no
  [then_run:...] marker of any kind appears.
• Same set minus pi-fabric: fusion returns, 5 / 5 again.

pi-fabric is a code-mode harness — it replaces the tool surface with one flat tool (fabric_exec) and edits go through
pi.edit(path, old, new) inside a sandboxed program. That surface has no then_run parameter, so the model cannot request
the fusion and thenRun is never set: the bare-return path, silently. I would fold that into your conflict list as a
distinct mode, because it is not a race between two mechanisms — an extension that owns the surface removes a parameter
Action Fusion depends on, which is indistinguishable from the #20 version regression until you bisect the extension set.

It is worth being precise about the blast radius, because I initially over-generalised this myself: of the four mechanisms
only Action Fusion is affected. Under the same flat-tool harness ObservationPack, Evidence-Preserving Reducer and Online
Context Compact all still fire (they hook the context projection and tool_result pipelines rather than a named tool). Full
per-mechanism matrix in #22.

Two notes on your numbers:

1. On finding 1 (then_run is model-opt-in): I could not reproduce the part where the model declines to volunteer it. Non-frontier
   backend, max thinking, no instruction mentioning then_run, 5 / 5 edits included it. So the gap you measured may be
   backend- or thinking-level-specific rather than a property of the mechanism. Either way the README would set expectations
   better if it labelled then_run as model-supplied and noted that a directive tool description (or a system-prompt hint) is
   what makes the fusion fire on weaker backends.
2. Your point 4 (no per-mechanism ablation) looks right, and the measurements above are a small example of why it matters: with
   the flat-tool harness loaded, Action Fusion's measured effect is exactly zero, not "+14%" — you cannot tell "no benefit"
   from "mechanism unreachable" without checking which tool was actually called. Testing one mechanism at a time is what
   separated those two cases here.