feat: Comparison viewer — granular component extraction & multi-model side-by-side

[Troubladore]

Summary

Build a comparison viewer for Inspect AI that enables side-by-side evaluation of 2-4 models on the same task, with synchronized sample navigation and drill-down. This requires extracting the current monolithic viewer into granular, composable React components.

This is a proposed upstream contribution to UKGovernmentBEIS/inspect_ai, addressing issue #1327.

Motivation

The Gap

No tool in the Inspect AI ecosystem (upstream or any of 426 forks) provides multi-model comparison. This is the #1 requested viewer feature:

• #1327 — "Side by side comparison of two models" (open, maintainer confirmed "on our todo list" but blocked on refactoring)
• #2176 — "Paired analysis between 2 models on the same benchmark" (closed, DIY only)
• #704 — "Frameworks for summarising and visualising inspect logs" (closed, no solution)

Maintainer dragonstyle (Charles Teague, Posit) said in UKGovernmentBEIS#1327: "Right now there isn't an automated way to do this (though it is on our todo list)... I think we'd still like to do it!" He noted being "right in the middle of quite a bit of refactoring of the main application logic which is necessary to tackle a feature like this."

Our Use Case

The Grounding Measure benchmark evaluates 17 embedding models across G1-G4 metrics on SimLex-999 (999 word pairs each). We need to:

• Compare how two models score on the same word pair (e.g., "old/new" — why does GPT-2 get 0.89 but BERT gets 0.72?)
• See side-by-side G1 drift patterns for specific words across models
• Drill from a cross-model ranking table into paired sample views

Current State of the Viewer Architecture

What Already Exists (Upstream)

The viewer was extracted as a React library in PR #2464 (merged Sep 2025) by revmischa (METR):

Package: @meridianlabs/log-viewer v0.0.11
Build: yarn build:lib → ES modules via Vite library mode
Exports from src/inspect_ai/_view/www/src/index.ts:

// Monolithic app component
export { App } from "./app/App";

// API factories
export { clientApi } from "./client/api/client-api";
export { viewServerApi as createViewServerApi } from "./client/api/view-server/api-view-server";

// Types
export type { ClientAPI, LogViewAPI, LogHandle, LogPreview, ... } from "./client/api/types";

// State
export { initializeStore } from "./state/store";

What's Missing for Comparison

1. Granular component exports. Only App is exported — no SampleViewer, ScoreTable, EventTimeline, LogHeader, TranscriptPanel. These exist internally but aren't individually importable.

2. Multi-instance store. initializeStore() creates a single Zustand store managing one active log. Comparison needs N stores (one per panel) or a store redesigned for multi-log state.

3. Synchronized navigation. No mechanism to link sample selection across panels (e.g., "when I click pair_042 in panel A, also navigate to pair_042 in panel B").

4. The ClientAPI is single-log-oriented. Methods like get_log_summary(log_file) work for one log. A comparison backend needs to scope API calls per panel.

METR's Performance Work (Reference)

METR's faber/viewer-performance-combined branch (39 ahead, active Feb 2026) has patterns we should adopt:

Pattern	Implementation	Relevance
Server-side sample loading	New /log-sample endpoint	Critical — comparison loads N samples simultaneously
Stale state prevention	Generation counter in sampleSlice.ts	Essential for multi-panel synchronized updates
JSON Web Worker	json-worker.ts offloads parsing	Needed when parsing N logs simultaneously
Message pool dedup	log/_pool.py + frontend resolution	Already merged upstream (PR UKGovernmentBEIS#3374, log format v3)

METR's Pending Performance PRs on Upstream

5 open PRs from sjawhar (UKGovernmentBEIS#3353-3357): virtualization, lazy-loading, minification, cache headers, sample loading optimization. These should land before or alongside comparison work.

Proposed Architecture

Phase 1: Granular Component Extraction

Extract these as individually importable components:

Component	Current Location	Props Interface
SampleViewer	Embedded in app routes	{sample: EvalSample, scores: Score[]}
ScorePanel	Part of sample view	{scores: Score[], metrics: Metric[]}
TranscriptPanel	TranscriptPanel.tsx	{events: Event[], transforms: Transform[]}
LogHeader	Part of log view	{log: EvalLog, metadata: Record}
SampleList	Part of log view	{samples: SampleSummary[], onSelect: (id) => void}

Each component should:

• Accept data as props (not read from global store)
• Optionally accept a store instance for state management
• Be styled independently (CSS modules or scoped styles)
• Work standalone or composed in the existing App

Phase 2: Multi-Instance Store

Refactor initializeStore() to support:

// Current: single global store
const store = initializeStore();

// Proposed: parameterized instances
const storeA = initializeStore({ id: 'panel-a', logFile: 'model_a.eval' });
const storeB = initializeStore({ id: 'panel-b', logFile: 'model_b.eval' });

// With optional synchronization
const syncController = createSyncController([storeA, storeB], {
  syncSampleSelection: true,  // navigate both panels together
  syncScroll: false,           // independent scrolling
});

Phase 3: Comparison Route

New /compare route in the viewer:

http://localhost:7575/#/compare?logs=model_a.eval,model_b.eval

Layout options:

• Side-by-side (2 panels): most common, synchronized sample navigation
• Grid (2×2): for 4-model comparison
• Diff view: single panel highlighting score differences

Features:

• Synchronized sample list — selecting a sample in one panel selects it in all
• Score comparison overlay — show delta between panels
• Filter to "disagreement samples" — where models diverge most
• Summary bar — aggregate score comparison across all samples

Phase 4: Wrapper Dashboard (Our Layer)

A lightweight dashboard (could be the existing reporting/ module) that:

• Shows the cross-model ranking table (from analysis.md data)
• Links to Inspect View's /compare route for any selected model pair
• Links to single-model Inspect View for individual drill-down
• Reads from benchmarks/results/partials/*.json for aggregate data

Key Decisions Needed

1. Contribution strategy: Fork + PR to upstream? Or sidecar package that imports @meridianlabs/log-viewer?
2. Component extraction depth: Full extraction (all components individually importable) or minimal (just what comparison needs)?
3. Store architecture: Multiple store instances vs. single store with panel namespacing?
4. Synchronization model: Tight (all panels always in sync) vs. loose (opt-in sync)?
5. Routing: New route in existing viewer vs. separate app that embeds viewer components?
6. Relationship to METR's performance PRs: Wait for perf(viewer): re-enable virtualization for all transcripts UKGovernmentBEIS/inspect_ai#3353-3357 to merge, or build on top of current main?

Ecosystem Context

Key People

Person	Role	Relevance
dragonstyle (Charles Teague)	Inspect maintainer, Posit	Gatekeeper, expressed interest in UKGovernmentBEIS#1327
jjallaire (JJ Allaire)	Inspect creator, Posit founder	Architecture decisions
revmischa (Mischa Spiegelmock)	METR, react-lib author	Did the initial extraction (PR UKGovernmentBEIS#2464)
rasmusfaber (Rasmus Faber)	METR, perf branch author	Performance patterns we'd build on
sjawhar	METR, 5 pending perf PRs	Viewer perf work, potential collaborator
yassersouri	Opened UKGovernmentBEIS#1327	Offered to contribute if given breakdown

Fork Landscape

• 426 forks, only METR does real viewer work
• No fork has attempted comparison views — completely unoccupied niche
• METR's viewer branches: 13+ branches, performance-focused, no comparison

Upstream Viewer Tech Stack

• React + TypeScript (strict mode)
• Zustand for state management (single store, slice pattern)
• Vite for build (dual app/library mode)
• react-virtuoso for large list rendering
• Dexie.js (IndexedDB) for client-side caching
• aiohttp Python server backend

Related Issues

• Depends on improved metrics value_to_float string conversion UKGovernmentBEIS/inspect_ai#196 (filename parsing fix — needed for our eval logs to load correctly)
• Extends the @meridianlabs/log-viewer library (upstream PR React Library + API Fetch Interface UKGovernmentBEIS/inspect_ai#2464)
• Addresses upstream Side by side comparison of two models UKGovernmentBEIS/inspect_ai#1327 (side-by-side comparison)
• Builds on METR's perf patterns (upstream PRs perf(viewer): re-enable virtualization for all transcripts UKGovernmentBEIS/inspect_ai#3353-3357)

Acceptance Criteria

• Design document with architecture decisions for all 6 key decisions above
• Proof of concept: two eval logs displayed side-by-side with synchronized sample selection
• Granular component exports added to @meridianlabs/log-viewer
• Multi-instance store working with independent panel state
• No regression in existing single-log viewer functionality
• PR submitted to upstream with tests and documentation
• Our benchmark dashboard links to comparison view for model pairs