docs: address reviewer methodology concerns

Author: jroell

README.md

@@ -13,7 +13,7 @@ The Agentic Epistemology Framework (AEF) provides a principled approach to model
 - **Facet Influence Ledger**: Records selected facets, prompt hashes, runtime weights, marker coverage, contradictions, and recommended debug actions.
 - **Counterfactual Diagnosis**: Supports facet removal/swap probes to estimate whether persona behavior changes in the expected direction.
 - **Observer Pattern**: Comprehensive tracing of epistemic events for transparency and debugging.
-- **Public Benchmarks**: Deterministic benchmark harnesses for root-cause diagnosis, facet-frame faithfulness, and the AEF audit service.
+- **Public Harnesses**: Deterministic trace-sensitivity harnesses for root-cause diagnosis, facet-frame faithfulness, and the AEF audit service.
 
 ## Installation
 
@@ -28,11 +28,11 @@ npm run build
 
 ## Getting Started
 
-Run the public benchmarks:
+Run the public harnesses:
 
 ```bash
 npx ts-node --transpile-only examples/root-cause-benchmark.ts --seed=42 --cases=160
-npx ts-node --transpile-only examples/facet-frame-faithfulness-benchmark.ts --seed=42 --cases=200
+npx ts-node --transpile-only examples/facet-frame-faithfulness-benchmark.ts --seed=42 --seed-count=20 --cases=200
 npx ts-node --transpile-only examples/facet-faithfulness-audit-experiment.ts
 ```
 

benchmarks/README.md

@@ -1,12 +1,12 @@
-# AEF Public Benchmarks
+# AEF Public Harnesses
 
-This directory documents the reproducible benchmark harnesses included with the Agentic Epistemology Framework.
+This directory documents the reproducible trace-sensitivity harnesses included with the Agentic Epistemology Framework.
 
-The benchmark code lives in `examples/` so it can run directly against the TypeScript source without a package publish step.
+The harness code lives in `examples/` so it can run directly against the TypeScript source without a package publish step.
 
-## Benchmarks
+## Harnesses
 
-### 1. Generic root-cause benchmark
+### 1. Generic root-cause harness
 
 Tests whether structured epistemic traces expose fields needed to localize injected agent failure labels in a deterministic harness.
 
@@ -28,14 +28,14 @@ results/root-cause-benchmark/
 └── accuracy-by-condition.svg
 ```
 
-Primary metric: diagnosis accuracy by trace condition.
+Primary metric: diagnosis accuracy by trace condition. This is an information-availability check, not an external-validity estimate.
 
-### 2. Facet-frame faithfulness benchmark
+### 2. Facet-frame faithfulness harness
 
-Tests whether facet-aware traces improve diagnosis of synthetic persona failures such as facet omission, facet misweighting, facet conflict, persona drift, and response-style violation.
+Tests whether facet-aware traces expose fields needed to localize synthetic persona failures such as facet omission, facet misweighting, facet conflict, persona drift, and response-style violation.
 
 ```bash
-npx ts-node --transpile-only examples/facet-frame-faithfulness-benchmark.ts --seed=42 --cases=200
+npx ts-node --transpile-only examples/facet-frame-faithfulness-benchmark.ts --seed=42 --seed-count=20 --cases=200
 ```
 
 Outputs:
@@ -46,15 +46,17 @@ results/facet-frame-faithfulness/
 ├── metrics.json
 ├── metrics.csv
 ├── metrics-table.tex
+├── metrics-summary.json
+├── metrics-summary-table.tex
 ├── qualitative-example.json
 └── accuracy-by-condition.svg
 ```
 
-Primary metric: diagnosis accuracy and macro F1 by trace condition.
+Primary metrics: diagnosis accuracy, macro F1, and coverage by trace condition.
 
-### 3. Facet faithfulness audit service experiment
+### 3. Facet faithfulness audit service case study
 
-Tests the implemented `FacetFaithfulnessAuditService` on Vurvey-style persona cases with prompt provenance, marker coverage, contradictions, counterfactual sensitivity, calibration, and signal-to-noise scoring.
+Tests the implemented `FacetFaithfulnessAuditService` on Vurvey-style persona fixtures with prompt provenance, marker coverage, contradictions, counterfactual sensitivity, and calibration scoring.
 
 ```bash
 npx ts-node --transpile-only examples/facet-faithfulness-audit-experiment.ts
@@ -68,20 +70,22 @@ results/facet-faithfulness-audit-experiment/
 ├── audits.json
 ├── metrics.json
 ├── metrics-table.tex
-└── diagnosis-accuracy.svg
+├── calibration.json
+├── calibration-table.tex
+└── calibration.svg
 ```
 
-Primary metric: diagnosis accuracy by evidence condition.
+Primary metric: diagnosis accuracy by evidence condition. The calibration artifacts are a fixture sanity check, not a deployment reliability diagram.
 
 ## Reproducibility notes
 
-- All public benchmark inputs are synthetic fixtures.
+- All public harness inputs are synthetic fixtures.
 - The harnesses require no Vurvey credentials, no private database, and no model API calls.
 - The Vurvey real-generation counterfactual results included in the paper are reported only as sanitized aggregates under `results/vurvey-*`; raw persona text and raw database fixture rows are intentionally excluded.
 
 ## Current reference results
 
-### Generic root-cause benchmark
+### Generic root-cause harness
 
 | Condition | Accuracy | Macro F1 | Coverage |
 |---|---:|---:|---:|
@@ -90,33 +94,32 @@ Primary metric: diagnosis accuracy by evidence condition.
 | no_justification | 0.500 | 0.500 | 0.500 |
 | event_only | 0.000 | 0.000 | 0.000 |
 
-### Facet-frame faithfulness benchmark
+### Facet-frame faithfulness harness
 
 | Condition | Accuracy | Macro F1 | Coverage |
 |---|---:|---:|---:|
-| full | 0.865 | 0.898 | 0.865 |
-| no_facet_weights | 0.665 | 0.678 | 0.735 |
+| full | 1.000 | 1.000 | 1.000 |
+| no_facet_weights | 0.800 | 0.772 | 0.866 |
 | no_facet_prompts | 0.400 | 0.400 | 0.400 |
 | event_only | 0.000 | 0.000 | 0.000 |
 
-### Facet faithfulness audit service experiment
+### Facet faithfulness audit service case study
 
 | Condition | Accuracy | Faithfulness | Marker Coverage | Counterfactual |
 |---|---:|---:|---:|---:|
-| prompt_only | 0.125 | 0.162 | 0.000 | 0.500 |
+| response_only | 0.125 | 0.162 | 0.000 | 0.500 |
 | facet_list | 0.125 | 0.382 | 0.000 | 0.500 |
-| facet_weights | 0.750 | 0.526 | 0.281 | 0.500 |
+| facet_markers_weights | 0.750 | 0.526 | 0.281 | 0.500 |
 | full_aef | 1.000 | 0.536 | 0.281 | 0.660 |
 
-## How to add a new benchmark
+## How to add a new harness
 
-1. Add a new `examples/<benchmark-name>.ts` runner.
+1. Add a new `examples/<harness-name>.ts` runner.
 2. Make it deterministic by accepting `--seed`.
-3. Write outputs under `results/<benchmark-name>/`.
+3. Write outputs under `results/<harness-name>/`.
 4. Include at least `cases.json`, `metrics.json`, and a qualitative example.
 5. Add a short section to this README with the command and expected outputs.
 
-
 ## Reviewer caveat
 
 These public harnesses are deterministic trace-sensitivity tests. They are not external-validity estimates for deployed personas. Use them to check that a trace schema carries enough information for a diagnoser, then validate real persona behavior with independent labels, noisy traces, and live model outputs.

examples/custom-frame-example.ts

@@ -14,7 +14,8 @@ import {
   FrameRegistry,
   createFrameConfig,
   getFrameSystemInfo,
-  initializeFrameSystem
+  initializeFrameSystem,
+  ObservationJustificationElement
 } from '../src/epistemic';
 
 // ============================================================================
@@ -179,7 +180,7 @@ class DevilsAdvocateFrame extends ComposableBaseFrame implements IDebateFrame {
     try {
       // Get base strength evaluation
       const baseStrength = await this.llmProvider.judgeEvidenceStrength(
-        { id: '', type: 'argument', content: argument, source: 'debate' },
+        new ObservationJustificationElement('debate', { argument }),
         debateContext.topic
       );
 
@@ -322,7 +323,8 @@ export function exampleUsage(): void {
 // ============================================================================
 
 // 1. Initialize the frame system (once at startup)
-import { initializeFrameSystem } from '../src/epistemic';
+import { initializeFrameSystem   ObservationJustificationElement
+} from '../src/epistemic';
 import { registerCustomFrames, createOptimismFrame } from './custom-frame-example';
 
 initializeFrameSystem();

examples/facet-faithfulness-audit-experiment.ts

@@ -12,7 +12,7 @@ type Failure =
   | 'response_style_violation'
   | 'low_counterfactual_sensitivity';
 
-type Condition = 'prompt_only' | 'facet_list' | 'facet_weights' | 'full_aef';
+type Condition = 'response_only' | 'facet_list' | 'facet_markers_weights' | 'full_aef';
 
 interface Scenario {
   id: string;
@@ -30,6 +30,14 @@ interface MetricRow {
   meanCalibrationScore: number;
 }
 
+interface CalibrationRow {
+  bin: string;
+  count: number;
+  meanConfidence: number;
+  meanFaithfulness: number;
+  absoluteGap: number;
+}
+
 const service = new FacetFaithfulnessAuditService();
 
 const crisisFacet = (): FacetAuditFacet => ({
@@ -203,7 +211,7 @@ function scenarios(): Scenario[] {
 
 function applyCondition(input: FacetAuditInput, condition: Condition): FacetAuditInput {
   if (condition === 'full_aef') return input;
-  if (condition === 'facet_weights') {
+  if (condition === 'facet_markers_weights') {
     return { ...input, renderedPrompt: undefined, counterfactuals: undefined };
   }
   if (condition === 'facet_list') {
@@ -234,10 +242,75 @@ function round(value: number): number {
   return Math.round(value * 1000) / 1000;
 }
 
+function calibrationRows(cases: Scenario[]): CalibrationRow[] {
+  const bins = [
+    { label: '0.00-0.33', min: 0, max: 0.33 },
+    { label: '0.34-0.66', min: 0.34, max: 0.66 },
+    { label: '0.67-1.00', min: 0.67, max: 1 },
+  ];
+  const audits = cases.map((item) => ({ item, audit: service.audit(item.input) }));
+  return bins.map((bin) => {
+    const matching = audits.filter(({ item }) => {
+      const confidence = item.input.selfReportedConfidence ?? 0;
+      return confidence >= bin.min && confidence <= bin.max;
+    });
+    if (matching.length === 0) {
+      return { bin: bin.label, count: 0, meanConfidence: 0, meanFaithfulness: 0, absoluteGap: 0 };
+    }
+    const meanConfidence = mean(matching.map(({ item }) => item.input.selfReportedConfidence ?? 0));
+    const meanFaithfulness = mean(matching.map(({ audit }) => audit.faithfulnessScore));
+    return {
+      bin: bin.label,
+      count: matching.length,
+      meanConfidence: round(meanConfidence),
+      meanFaithfulness: round(meanFaithfulness),
+      absoluteGap: round(Math.abs(meanConfidence - meanFaithfulness)),
+    };
+  });
+}
+
+function calibrationLatex(rows: CalibrationRow[]): string {
+  return [
+    '\\begin{tabular}{lrrrr}',
+    '\\toprule',
+    'Confidence bin & Count & Mean confidence & Mean faithfulness & Gap \\\\',
+    '\\midrule',
+    ...rows.map((row) => `${row.bin} & ${row.count} & ${row.meanConfidence.toFixed(3)} & ${row.meanFaithfulness.toFixed(3)} & ${row.absoluteGap.toFixed(3)} \\\\`),
+    '\\bottomrule',
+    '\\end{tabular}',
+    '',
+  ].join('\n');
+}
+
+function writeCalibrationFigure(rows: CalibrationRow[], outputPath: string): void {
+  const width = 640;
+  const height = 420;
+  const margin = { left: 80, right: 40, top: 40, bottom: 70 };
+  const plotWidth = width - margin.left - margin.right;
+  const plotHeight = height - margin.top - margin.bottom;
+  const points = rows.filter((row) => row.count > 0).map((row) => {
+    const x = margin.left + row.meanConfidence * plotWidth;
+    const y = margin.top + plotHeight - row.meanFaithfulness * plotHeight;
+    return `<circle cx="${x}" cy="${y}" r="8" fill="#0072B2" />\n<text x="${x + 10}" y="${y - 10}" font-size="12">${row.bin} (n=${row.count})</text>`;
+  }).join('\n');
+  const svg = `<svg xmlns="http://www.w3.org/2000/svg" width="${width}" height="${height}" viewBox="0 0 ${width} ${height}">
+  <rect width="100%" height="100%" fill="white"/>
+  <text x="${width / 2}" y="24" text-anchor="middle" font-size="18" font-family="Arial">Calibration: confidence vs. measured faithfulness</text>
+  <line x1="${margin.left}" y1="${margin.top + plotHeight}" x2="${width - margin.right}" y2="${margin.top}" stroke="#999" stroke-dasharray="5,5" />
+  <line x1="${margin.left}" y1="${margin.top}" x2="${margin.left}" y2="${margin.top + plotHeight}" stroke="#333" />
+  <line x1="${margin.left}" y1="${margin.top + plotHeight}" x2="${width - margin.right}" y2="${margin.top + plotHeight}" stroke="#333" />
+  <text x="${margin.left + plotWidth / 2}" y="${height - 25}" text-anchor="middle" font-size="14">Self-reported confidence</text>
+  <text transform="translate(24,${margin.top + plotHeight / 2}) rotate(-90)" text-anchor="middle" font-size="14">Measured faithfulness</text>
+  ${points}
+</svg>\n`;
+  fs.writeFileSync(outputPath, svg);
+}
+
 
 function run(): void {
   const cases = scenarios();
-  const conditions: Condition[] = ['prompt_only', 'facet_list', 'facet_weights', 'full_aef'];
+  const calibration = calibrationRows(cases);
+  const conditions: Condition[] = ['response_only', 'facet_list', 'facet_markers_weights', 'full_aef'];
   const metrics: MetricRow[] = conditions.map((condition) => {
     const results = cases.map((item) => {
       const audit = service.audit(applyCondition(item.input, condition));
@@ -260,6 +333,9 @@ function run(): void {
   fs.writeFileSync(path.join(outDir, 'metrics.json'), JSON.stringify(metrics, null, 2));
   fs.writeFileSync(path.join(outDir, 'cases.json'), JSON.stringify(cases, null, 2));
   fs.writeFileSync(path.join(outDir, 'audits.json'), JSON.stringify(cases.map((item) => ({ id: item.id, expectedFailure: item.expectedFailure, audit: service.audit(item.input) })), null, 2));
+  fs.writeFileSync(path.join(outDir, 'calibration.json'), JSON.stringify(calibration, null, 2));
+  fs.writeFileSync(path.join(outDir, 'calibration-table.tex'), calibrationLatex(calibration));
+  writeCalibrationFigure(calibration, path.join(outDir, 'calibration.svg'));
   fs.writeFileSync(path.join(outDir, 'metrics-table.tex'), [
     '\\begin{tabular}{lrrrrr}',
     '\\toprule',
@@ -271,7 +347,7 @@ function run(): void {
     '',
   ].join('\n'));
 
-  console.log(JSON.stringify({ metrics, outDir }, null, 2));
+  console.log(JSON.stringify({ metrics, calibration, outDir }, null, 2));
 }
 
 run();