fix(harness): serve the system prompt reliably — SYSTEM_MD env-forward + materialization + observability#76
Open
sunholo-voight-kampff wants to merge 6 commits into
Conversation
…ls counting + output headroom + local summarizer Squashed reliable-compaction work. Resolves the chars/4 over-count thrash and the dead cloud-summarizer, on top of arniwesth#70: - token estimate counts tool_calls args and uses calibrated chars/7 (measured ~7.5 chars/tok for AILANG code + DOCX XML; old chars/4 over-counted ~1.9x -> premature compaction -> working-set thrash) - reserve 75k output headroom so input can't crowd the context window - compact_step_actual drives the elision LEVEL off the provider's ACTUAL last input_tokens (ground truth), not a char estimate; LoopTotals.last_input_tokens - wire compaction_ai's summarizer to a local ollama model (registry default points at a cloud model that's dead on a local-only rig) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The system message (AILANG reference) was seeded into the conversation list and flowed through compaction, which could drop/reorder it — so the model lost its syntax reference after compaction and drifted into invalid syntax (writing `module a.b.c` / `use std.xml`, i.e. another language). partition_system_msgs() splits ALL system messages off before BOTH compaction layers and re-prepends them for every provider call, so the reference is structurally immune to compaction. Verified: docx reimplement 0/17 -> 16/17, markdown 3/3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… failure The AILANG language reference was silently never reaching the model. The ollama profile config.json sets system_prompt="", which overrode the SYSTEM_MD env that run.sh sets, so cfg.agent.system_prompt resolved to "" -> raw_system="" -> the model got a 0-char system prompt: neither the harness instructions NOR the AILANG reference. It wrote AILANG from training + reading project files, and docx ground to 0/17 at max_steps. This stayed invisible for days because the served system prompt was never logged. Three changes, all in rpc.ail run_with_config: - env-authoritative: SYSTEM_MD env now wins over the profile config (one explicit source of truth, env over config) so the reference actually reaches the model. - observability: emit `system_prompt_built` every run (source, chars, has_ailang_reference, preview). "Is the reference served?" is now one log line, not a multi-day investigation. - loud failure (CLAUDE.md rule #2: no silent fallbacks): emit an `error` event when a system-prompt source was set but produced empty content. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rpc.ail env-authoritative read (f9f722e) couldn't work because the .ail runs as a subprocess spawned with an EXPLICIT env allowlist (runtime-process.ts childEnv) that scrubs every unlisted var. SYSTEM_MD wasn't listed, so getEnvOr("SYSTEM_MD") was always "" inside the .ail — the model ran with a 0-char system prompt (no harness instructions, no AILANG reference). docx 0/17. Same gotcha the file already documents for MOTOKO_PERSIST_RETRIES / MOTOKO_REPO. Add SYSTEM_MD to the allowlist. Now the system_prompt_built event reports the reference as served. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… the scrub-bug class) The .ail runtime subprocess is spawned with a hand-maintained env allowlist (childEnv). Any var not listed is silently scrubbed — which has broken at least FIVE env-gated features whose var someone forgot to add: SYSTEM_MD (4-day system-prompt bug), MOTOKO_REPO, MOTOKO_PERSIST_RETRIES, AILANG_OLLAMA_MAX_TOKENS, AILANG_STDLIB_PATH. AILANG_OLLAMA_MAX_TOKENS was even hand-added twice (a TS1117 duplicate-key error), proving the list is unmaintainable. Root cause: the allowlist is a second source of truth that drifts from config.ts's CORE_MAP — which already maps SYSTEM_MD. Systemic fix (audit-before-patch, not a 6th one-off): - autoForwardedEnvKeys(): derive the forward set from CORE_MAP + EXTENSION_MAPS + the MOTOKO_*/AILANG_* namespaces + SYSTEM_MD. New env-gated features now reach the .ail BY DEFAULT instead of failing silently. Explicit childEnv entries still win (computed values/defaults); arbitrary host vars are NOT forwarded (no secret leak). - Remove the duplicate AILANG_OLLAMA_MAX_TOKENS block (the TS1117 error). - runtime-process-env.test.ts: drift guard asserting forwarding covers every CORE_MAP /EXTENSION_MAPS var — fails loudly if the allowlist ever stops covering a config key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…space The AILANG runtime is FS-sandboxed to workdir (AILANG_FS_SANDBOX), so a SYSTEM_MD that points outside it (e.g. an eval harness writing the prompt to /tmp) is silently unreadable — rpc.ail's readFile returns empty and the model runs with no system prompt. The harness already materializes external --system-prompt files into the workspace; apply the same treatment to SYSTEM_MD. Now ANY caller's external prompt path just works, instead of each of the 5 eval scripts (and every future one) having to remember to place the file inside the workspace. The loud system_prompt_built event still surfaces any remaining gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The AILANG runtime subprocess is spawned with an explicit env allowlist (
runtime-process.tschildEnv). Any var not listed is silently scrubbed — andSYSTEM_MD(the path to the system-prompt file, e.g. the AILANG language reference served to agent evals) was not listed. SogetEnvOr("SYSTEM_MD")was always""inside the.ail, the system prompt resolved to 0 chars, and the model ran with no system prompt at all — no harness instructions, no language reference. It stayed invisible because the served system prompt was never logged.This is the same scrub-bug class the file already documents for
MOTOKO_REPO,MOTOKO_PERSIST_RETRIES,AILANG_OLLAMA_MAX_TOKENS,AILANG_STDLIB_PATH—SYSTEM_MDwas the 5th instance (AILANG_OLLAMA_MAX_TOKENSwas even hand-added twice).Fix (the 4 commits in this PR)
f9f722erpc.ail: env-authoritative system prompt + asystem_prompt_builtobservability event (source / chars /has_ailang_reference/ preview) + a loudSYSTEM PROMPT EMPTYerror. The observability turned a multi-day mystery into a one-line log.ec90f22runtime-process.ts: forwardSYSTEM_MDthrough the allowlist.c5b0924runtime-process.ts: systemic — derive the subprocess env allowlist fromconfig.tsCORE_MAP+EXTENSION_MAPS+ theMOTOKO_*/AILANG_*namespaces, so a new env-gated feature reaches the.ailby default instead of failing silently. Adds the drift-guard testruntime-process-env.test.ts; removes a duplicate-key error.b58b0d3index.ts: materialize an out-of-workspaceSYSTEM_MDinto the workspace (reusingmaterializeSystemPromptArg), since the FS sandbox blocks reads outsideworkdir.Proof: with the fixes,
system_prompt_builtreportschars=89825, has_ailang_reference=true(waschars=0); a docx agent eval'scat-the-full-file fallbacks dropped 11→8.This branch is cut off
fix/reliable-compaction(#75), so the diff vsmaincurrently also shows #75's compaction commits. Please review the last 4 commits (f9f722e..b58b0d3) — those are this PR. Once #75 merges, this diff reduces to just the system-prompt fixes.