fix(estimate): count every attempt, and price runtime by prompt size

Review follow-ups. Both blocking items understated cost in the same
direction, and both are fixed from data the run records already carry.

Output tokens counted only the final attempt while prompt tokens were
multiplied by the attempt count, so a retried pass recorded prompt x2 and
output x1. Because calibrate divides output by a request count that
includes retries, every retry dragged outputTokensPerRequest down and
estimate then understated output for everyone.

runAuditWithAutoFix now reports promptChars and outputChars accumulated
across attempts, so both sides are exact rather than the final attempt
scaled up: the retry's prompt carries a correction section the x2
approximation missed, and its discarded first response cost tokens to
generate. Characters rather than tokens keeps the approximation in one
place, and keeps whole prompts from being retained on the result.

Runtime was requests x a flat msPerRequest, which is prompt-size-blind —
the exact case the command is pitched at. Calibration now carries a fixed
msPerRequest plus a marginal msPerPromptToken, least-squares fitted
across records when they differ in prompt size. When they cannot separate
the terms — one record, or every run the same size — the measured average
is split using the proportion the defaults imply, so the magnitude stays
measured even where the shape is assumed. A negative fitted term falls
back the same way. The defaults still sum to the previous 45s at a
2,500-token prompt.

Also:

- The --clone warning selected on the post-applies_to count, so a repo
  that was checked out but matched no files was told to clone what it
  already had. RepoEstimate carries the pre-scoping count and the two
  cases now get separate warnings; the second is the more useful, since
  it means a pack is pointed at a repo it cannot see.
- Docs and --help claimed the command "mutates nothing" while documenting
  --clone. Reworded, and the runtime model is documented.
- formatDuration handed over to minutes at 90s while rounding minutes
  from 60s, so "1 minute(s)" was unreachable: 89s rendered as seconds and
  90s jumped to 2 minutes.
- runEstimate printed targetErrors to stderr even when the report went to
  stdout, where the caveats already list them. Now only when --output
  redirects the report to a file. The 0 exit is left as it was, with a
  comment marking it deliberate rather than an oversight.

Left alone: the unguarded readFileSync(configPath). It matches runRun, so
fixing one without the other would just make the two inconsistent.

Breaking change, from the original commit rather than this one:
PackOutcome.usage is required and PackOutcome is exported from
source/index.ts, so an external consumer constructing one breaks at
compile time.

Author: addyCooks

docs/cli/index.md

@@ -60,7 +60,7 @@ The same validator, dedup logic, and findings model apply in both contexts, so w
 
 ## `estimate`
 
-Sizes an audit **before** it runs: how many repositories and rule packs are in scope, how many files they put in front of the model, and roughly how many model requests, tokens, and minutes that costs. It runs no model, files nothing, and mutates nothing — useful when you are about to point Sentinel at a dozen more repositories, or adding a pack to every target and want to know what that does to the nightly window.
+Sizes an audit **before** it runs: how many repositories and rule packs are in scope, how many files they put in front of the model, and roughly how many model requests, tokens, and minutes that costs. It runs no model and files no issues — `--clone` is the one flag that writes anything, checking out missing repos. Useful when you are about to point Sentinel at a dozen more repositories, or adding a pack to every target and want to know what that does to the nightly window.
 
 ```bash
 npx @nanocollective/sentinel estimate
@@ -90,8 +90,10 @@ Calibrated from the last 6 run record(s).
 
 ### How the figures are produced
 
-The token figure is **measured, not guessed**: `estimate` assembles the same prompts the audit would send — the pack body, the reporting contract, and the source files scoped by each pack's `applies_to.paths` — and counts them. What varies between installs is the per-request cost, so the request, token, and runtime figures are calibrated from the run records the last ten runs committed. Every run is instrumented for this: it records how long each pack pass took, how many model requests it made (auto-fix retries included), and the tokens it sent and received.
+The token figure is **measured, not guessed**: `estimate` assembles the same prompts the audit would send — the pack body, the reporting contract, and the source files scoped by each pack's `applies_to.paths` — and counts them. What varies between installs is the per-request cost, so the request, token, and runtime figures are calibrated from the run records the last ten runs committed. Every run is instrumented for this: it records how long each pack pass took, how many model requests it made (auto-fix retries included), and the tokens it sent and received across every attempt — an auto-fix retry resends the prompt and generates a second response, and both are counted.
+
+Runtime is **not** a flat per-request average. A request costs a fixed amount regardless of size plus an amount that tracks prompt size, and both terms are fitted from the records, so sizing a config far larger than anything you have run is not priced as though the prompts stayed the same. When the records cannot separate the two — a single run, or every run the same size — the measured average is split using the proportion the built-in defaults imply.
 
 Until a run has been recorded, the figures fall back to built-in defaults, and the output says so. Treat a first, uncalibrated estimate as an order of magnitude rather than a number to schedule against.
 
-Repositories already checked out under `--workspace` are measured from their real files. Any that are not are counted with zero files and called out in the output, so a partial estimate never reads as the whole picture — pass `--clone` to check the rest out first.
+Repositories already checked out under `--workspace` are measured from their real files. Any that are not are counted with zero files and called out in the output, so a partial estimate never reads as the whole picture — pass `--clone` to check the rest out first. A repo that *is* checked out but whose files no pack matches gets a separate warning: nothing needs cloning, but a pack is pointed at a repository it cannot see.

source/cli.ts

@@ -338,8 +338,8 @@ async function runRun(argv: string[]): Promise<number> {
 const ESTIMATE_USAGE = `sentinel estimate [options]
 
 Size an audit before running it: repositories, rule packs, files, model
-requests, tokens, and wall-clock runtime. Runs no model, files nothing, and
-mutates nothing.
+requests, tokens, and wall-clock runtime. Runs no model and files no issues;
+--clone is the one flag that writes anything, checking out missing repos.
 
 Figures are calibrated from the committed run records when any exist, so they
 sharpen against your own hardware and model. Repos already checked out under
@@ -386,10 +386,17 @@ async function runEstimate(argv: string[]): Promise<number> {
 		},
 	);
 
-	writeReport(renderEstimate(estimate), flagStr(flags, 'output'));
-	for (const error of estimate.targetErrors) {
-		console.error(`target: ${error}`);
+	const output = flagStr(flags, 'output');
+	writeReport(renderEstimate(estimate), output);
+	// The estimate already renders these as caveats, so repeat them on stderr
+	// only when the report went to a file and nobody would otherwise see them.
+	if (output) {
+		for (const error of estimate.targetErrors) {
+			console.error(`target: ${error}`);
+		}
 	}
+	// Deliberately 0 even when targets failed: estimate is advisory, and a
+	// partial estimate is still useful — the caveats say what is missing.
 	return 0;
 }
 

source/orchestrator/auto-fix.spec.ts

@@ -129,3 +129,44 @@ test('buildAutoFixPrompt renders document-level errors without an index', t => {
 	t.true(prompt.includes('- document: no array found'));
 	t.false(prompt.includes('finding[-1]'));
 });
+
+// The audit layer turns these into the prompt/output token figures `estimate`
+// calibrates from, so they have to cover every attempt rather than the last.
+test('reports the characters sent and received on a single attempt', async t => {
+	const output = JSON.stringify([GOOD]);
+	const runner = queuedRunner([{ok: true, output}]);
+	const result = await runAuditWithAutoFix('prompt', MODEL, runner);
+	t.is(result.promptChars, 'prompt'.length);
+	t.is(result.outputChars, output.length);
+});
+
+test('accumulates characters across a retry', async t => {
+	const first = JSON.stringify([BAD]);
+	const second = JSON.stringify([GOOD]);
+	const runner = queuedRunner([
+		{ok: true, output: first},
+		{ok: true, output: second},
+	]);
+	const result = await runAuditWithAutoFix('prompt', MODEL, runner);
+
+	t.is(result.attempts, 2);
+	// Both responses counted, not just the one that validated.
+	t.is(result.outputChars, first.length + second.length);
+	// Both prompts counted, and the retry carried the correction section, so
+	// the total exceeds twice the original rather than matching it.
+	t.is(
+		result.promptChars,
+		(runner.prompts[0]?.length ?? 0) + (runner.prompts[1]?.length ?? 0),
+	);
+	t.true(result.promptChars > 'prompt'.length * 2);
+});
+
+test('a process failure is not retried and counts one exchange', async t => {
+	const runner = queuedRunner([
+		{ok: false, output: '', error: 'nanocoder missing'},
+	]);
+	const result = await runAuditWithAutoFix('prompt', MODEL, runner);
+	t.is(result.attempts, 1);
+	t.is(result.promptChars, 'prompt'.length);
+	t.is(result.outputChars, 0);
+});

source/orchestrator/auto-fix.ts

@@ -22,6 +22,15 @@ export interface AutoFixOptions extends RunnerOptions {
 export interface AutoFixResult extends AuditResult {
 	/** Number of model runs performed (1 = succeeded or failed on first try). */
 	attempts: number;
+	/**
+	 * Characters sent across every attempt. A retry resends the audit prompt plus
+	 * a correction section, so this is the real total rather than a multiple of
+	 * the first prompt. Carried as characters, not tokens, to keep the token
+	 * approximation in one place ({@link ../run/estimate.js estimateTokens}).
+	 */
+	promptChars: number;
+	/** Characters returned across every attempt, the discarded ones included. */
+	outputChars: number;
 }
 
 function formatErrors(errors: ValidationError[]): string {
@@ -83,12 +92,16 @@ export async function runAuditWithAutoFix(
 
 	let result = await runAudit(prompt, model, runner, options);
 	let attempts = 1;
+	let promptChars = prompt.length;
+	let outputChars = result.raw.length;
 
 	while (!result.ok && !result.runError && attempts < maxAttempts) {
 		const fixPrompt = buildAutoFixPrompt(prompt, result);
 		result = await runAudit(fixPrompt, model, runner, options);
 		attempts++;
+		promptChars += fixPrompt.length;
+		outputChars += result.raw.length;
 	}
 
-	return {...result, attempts};
+	return {...result, attempts, promptChars, outputChars};
 }

source/run/audit.spec.ts

@@ -3,6 +3,7 @@ import type {ModelConfig} from '../config/types.js';
 import type {ModelRunner, ModelRunResult} from '../orchestrator/types.js';
 import type {RulePack} from '../rule-packs/types.js';
 import {auditPack} from './audit.js';
+import {estimateTokens} from './estimate.js';
 
 console.log('\nrun/audit.spec.ts');
 
@@ -70,22 +71,44 @@ test('measures the pass so estimates have something to calibrate on', async t =>
 	t.true(outcome.usage.outputTokens > 0);
 });
 
-test('counts the prompt once per attempt', async t => {
-	// Malformed output the auto-fix loop retries, so two attempts are made.
-	const single = await auditPack(
+test('counts every attempt on both the prompt and the output side', async t => {
+	// The same malformed output, run once and then with the retry allowed, so
+	// the only difference between the two is the second attempt.
+	const context = {repoName: 'org/a', files: []};
+	const once = await auditPack(
 		PACK,
-		{repoName: 'org/a', files: []},
+		context,
 		MODEL,
-		runner({ok: true, output: JSON.stringify([FINDING])}),
+		runner({ok: true, output: 'not json'}),
+		{maxAttempts: 1},
 	);
 	const retried = await auditPack(
 		PACK,
-		{repoName: 'org/a', files: []},
+		context,
 		MODEL,
 		runner({ok: true, output: 'not json'}),
 	);
+
+	t.is(once.attempts, 1);
 	t.is(retried.attempts, 2);
-	t.is(retried.usage.promptTokens, single.usage.promptTokens * 2);
+	// The discarded first response still cost tokens to generate. Counting only
+	// the final attempt understated output, and since calibration divides by a
+	// request count that includes retries, it dragged every estimate down.
+	t.is(retried.usage.outputTokens, once.usage.outputTokens * 2);
+	// The retry resends the prompt plus a correction section, so the prompt side
+	// is more than twice the first attempt rather than exactly twice.
+	t.true(retried.usage.promptTokens > once.usage.promptTokens * 2);
+});
+
+test('a pass that never retries counts one prompt and one response', async t => {
+	const outcome = await auditPack(
+		PACK,
+		{repoName: 'org/a', files: []},
+		MODEL,
+		runner({ok: true, output: JSON.stringify([FINDING])}),
+	);
+	t.is(outcome.attempts, 1);
+	t.is(outcome.usage.outputTokens, estimateTokens(JSON.stringify([FINDING])));
 });
 
 test('surfaces a run error in the outcome', async t => {

source/run/audit.ts

@@ -14,7 +14,7 @@ import type {ModelRunner} from '../orchestrator/types.js';
 import {buildAuditPrompt} from '../prompt/build.js';
 import type {SourceFile} from '../prompt/types.js';
 import type {RulePack} from '../rule-packs/types.js';
-import {estimateTokens} from './estimate.js';
+import {tokensFromChars} from './estimate.js';
 import type {PackOutcome} from './types.js';
 
 /** The repository material one pack pass audits. */
@@ -56,10 +56,12 @@ export async function auditPack(
 		raw: result.raw,
 		usage: {
 			durationMs,
-			// A retry resends the audit prompt; the correction preamble is small
-			// beside it, so attempts x the base prompt is a fair figure.
-			promptTokens: estimateTokens(prompt) * result.attempts,
-			outputTokens: estimateTokens(result.raw),
+			// Both sides are the real totals across every attempt, not the final
+			// one scaled up: a retry resends the prompt plus a correction section,
+			// and the output it discards still cost tokens to generate. Counting
+			// them keeps calibration honest, since requests includes retries too.
+			promptTokens: tokensFromChars(result.promptChars),
+			outputTokens: tokensFromChars(result.outputChars),
 		},
 	};
 }