Add Persistence View, multi-turn dataset examples, and project hygiene

HTML report: replace Flow tab with Persistence View (heatmap of content
blocks × requests, blue intensity = token cost, COMPACT? annotations).
Flow code retained but hidden; Artifacts pane unchanged.

Multi-turn examples: add SWE-agent, lmcache, OpenHands trajectory demos
with conversion scripts and pre-converted session fixtures. README gains
a Findings Across Public Datasets table (3 datasets, real profiler output).

Project hygiene: remove toolathon fixtures and GIFs; refresh screenshots
to match new views; update CLAUDE.md with accurate architecture, format
strategy, and HTML view names; update skill with missing issue codes;
remove planned OTel from plugin manifests; add .gitignore entries.

Author: yanpgwang

.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
   "plugins": [
     {
       "name": "context-profiler",
-      "description": "Analyze agent traces, transcripts, Langfuse exports, OpenTelemetry spans, and raw provider requests with context-profiler.",
+      "description": "Analyze agent traces, transcripts, Langfuse exports, and raw provider requests with context-profiler.",
       "source": "./",
       "strict": false,
       "skills": [

.gitignore

@@ -41,9 +41,11 @@ trace-*.json
 .claude/
 .cursor/
 
+# Playwright MCP artifacts
+.playwright-mcp/
+
 # uv
 uv.lock
 
 # Example generated files (keep raw data, ignore generated)
 examples/report.html
-examples/session.jsonl

.plugin/plugin.json

@@ -2,6 +2,7 @@
   "name": "context-profiler",
   "description": "Agent context analysis harness for diagnosing context growth, tool bloat, repeated content, and turn-to-turn trace evolution.",
   "version": "0.1.0",
+  "repository": "https://github.com/Turdot/context-profiler",
   "author": {
     "name": "Yanpeng Wang"
   },

CLAUDE.md

@@ -4,9 +4,9 @@ This file guides AI coding agents working on `context-profiler`.
 
 ## Project Overview
 
-`context-profiler` is a trace-source agnostic context analysis harness for LLM agents. It accepts raw provider requests, observability traces, local coding-agent transcripts, and future benchmark trajectories, then reports how context grows, repeats, and concentrates around tools/artifacts.
+`context-profiler` is the evidence layer for context engineering. It accepts raw provider requests, observability traces, and agent trajectories, then reports how context grows, repeats, and concentrates around tools/artifacts — so humans and agents know what to compact and where it's safe to cut.
 
-Core boundary: this project analyzes traces. It does not fetch traces, replay agent loops, or execute third-party tools.
+Core boundary: the library and CLI analyze trace files or stdin. Agents may fetch user-requested trace exports with tools such as `curl` before invoking `context-profiler`, but core should not replay agent loops or execute third-party tools.
 
 ## Common Commands
 
@@ -17,29 +17,47 @@ PYTHONPATH=src uv run --with pytest pytest tests/test_smoke.py -v
 # Check CLI entry point
 PYTHONPATH=src uv run context-profiler --help
 
-# Analyze a raw request
-PYTHONPATH=src uv run context-profiler analyze tests/fixtures/repeated_tool_calls.json --format openai
+# Analyze a multi-turn session (canonical demo)
+PYTHONPATH=src uv run context-profiler analyze examples/swe_agent/session.jsonl --format openai --html /tmp/report.html
 
-# Diagnose a transcript
-PYTHONPATH=src uv run context-profiler diagnose tests/fixtures/cursor_transcript.jsonl --format cursor-jsonl --json
+# Diagnose for agent consumption
+PYTHONPATH=src uv run context-profiler diagnose examples/swe_agent/session.jsonl --format openai --json
 
-# Generate HTML
-PYTHONPATH=src uv run context-profiler analyze tests/fixtures/cursor_transcript.jsonl --format cursor-jsonl --html /tmp/context-profiler-report.html
+# Analyze a raw request
+PYTHONPATH=src uv run context-profiler analyze tests/fixtures/repeated_tool_calls.json --format openai
 ```
 
 ## Architecture
 
 - `src/context_profiler/cli.py` — Click CLI entry point.
+- `src/context_profiler/models.py` — core data model (Session, APIRequest, Message, ContentBlock, BlockType, Role).
+- `src/context_profiler/adapters/` — format-specific input adapters (OpenAI, Anthropic, Langfuse, agent-trace, transcript).
 - `src/context_profiler/formats.py` — format registry exposed to agents.
 - `src/context_profiler/schemas.py` — JSON Schemas for agent-readable contracts.
 - `src/context_profiler/validation.py` — validation and canonical normalization helpers.
 - `src/context_profiler/profiler.py` — session/request loading and analyzer orchestration.
 - `src/context_profiler/analyzers/` — token, repeat, and field analyzers.
 - `src/context_profiler/context_diff.py` — turn-to-turn diff evidence and hints.
+- `src/context_profiler/session_insights.py` — session-level carryover, budget pressure, artifact lifecycle, and propagation analysis.
 - `src/context_profiler/diagnostics.py` — stable issue-code diagnosis JSON.
 - `src/context_profiler/reporters/` — CLI, JSON, and HTML reports.
-- `src/context_profiler/templates/report.html` — self-contained report UI.
+- `src/context_profiler/templates/report.html` — self-contained report UI (views: Icicle, Tools, Persistence).
 - `skills/` — public Agent Skills distribution.
+- `examples/` — multi-turn demo datasets with conversion scripts.
+
+## HTML Report Views
+
+The interactive HTML report has three main views:
+
+- **Icicle** — token distribution per request, timeline navigation, semantic/diff color modes.
+- **Tools** — tool token hotspots, invocation detail, schema size.
+- **Persistence** — heatmap of content blocks × requests showing what persists across turns (replaces the old Flow view).
+
+## Format Strategy
+
+Core adapters (tested, maintained): `openai`, `anthropic`, `langfuse`, `cursor-jsonl`, `claude-code-jsonl`, `agent-trace`.
+
+For new benchmark trajectory formats (SWE-agent, T1, tau-bench, etc.), write conversion scripts in `examples/<dataset>/convert.py` that output OpenAI-format JSONL. Do not add new core adapters for every dataset.
 
 ## Format and Analyzer Rules
 
@@ -51,9 +69,9 @@ PYTHONPATH=src uv run context-profiler analyze tests/fixtures/cursor_transcript.
 
 ## Product Boundaries
 
-Do not add provider-specific fetch clients to core. Agents can use Langfuse public API exports via `curl`, SDKs, Hugging Face tooling, or local transcript discovery to obtain data, then pass files or stdin to `context-profiler`. Prefer direct Langfuse public API exports over `langfuse-cli` for trace analysis because the CLI can omit fields needed for nested observations and generations.
+Do not add provider-specific fetch clients to core. Agents can use Langfuse public API exports via `curl`, SDKs, Hugging Face tooling, or local transcript discovery to obtain data when the user asks, then pass files or stdin to `context-profiler`. Prefer direct Langfuse public API exports over `langfuse-cli` for trace analysis because the CLI can omit fields needed for nested observations and generations.
 
-Do not make Toolathlon a first-class multi-turn demo. Prefer genuinely multi-turn datasets such as `pagarsky/agent-trace`, `cx-cmu/agent_trajectories`, and SWE-agent trajectories for future research examples.
+Canonical multi-turn demo datasets are in `examples/`: SWE-agent trajectories (nebius), lmcache-agentic-traces (sammshen), and OpenHands trajectories (nvidia). For new datasets, write conversion scripts in `examples/<dataset>/convert.py` rather than adding core adapters.
 
 ## Skill Distribution
 

README.md

@@ -4,45 +4,75 @@
 [![Python](https://img.shields.io/pypi/pyversions/context-profiler)](https://pypi.org/project/context-profiler/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Trace-source agnostic context analysis harness for LLM agents.
+The evidence layer for context engineering. Profile before you prune.
 
-Bring any agent trace, loop, transcript, or raw provider request. `context-profiler` explains how the context grows, which tools dominate it, what repeats, and where turn-to-turn changes happen.
+`context-profiler` turns raw provider requests, observability exports, and agent trajectories into evidence about how context grows, repeats, and concentrates — so you know **what** to compact and **where** it's safe to cut.
 
-![Icicle view — token distribution breakdown](assets/demo-snapshot.gif)
+<p align="center">
+  <img src="assets/readme/hero.png" alt="context-profiler: Icicle view showing token distribution across a 31-turn SWE-agent session" width="100%">
+</p>
 
 ## Why
 
-Agent systems fail quietly when context grows too large, repeats stale observations, carries verbose tool payloads, or churns on the same artifact for many turns. Observability tools tell you what happened. `context-profiler` focuses on what happened to the context.
+Compression tools (LLMLingua, `/compact`, Mem0) execute blindly. They don't tell you what's redundant, what's safe to remove, or what downstream references will break. `context-profiler` fills the missing step:
 
-It is designed for both humans and agents:
-
-- Humans get an interactive HTML report with timeline, icicle, diff, and tool views.
-- Agents get stable JSON from `validate`, `diagnose`, and `schema` commands.
-- Skills teach Cursor, Claude Code, and other Agent Skills compatible tools when to call the CLI.
-
-![Session mode — timeline and diff](assets/demo-session.gif)
-
-## What It Finds
-
-- **Tool bloat**: tool inputs/results dominate the visible context.
-- **Context growth spikes**: individual turns that add large payloads.
-- **Repeated content**: exact or near-duplicate blocks retained in context.
-- **Repeated tool inputs**: large repeated tool arguments.
-- **Artifact churn**: the same file/component/path appears across many tool calls.
-- **Partial-scope warnings**: transcripts are useful, but not raw provider requests.
-
-## Research Context
-
-`context-profiler` is motivated by recent work showing that long-horizon agents are constrained not only by model quality, but also by how their working context is retained, compressed, and reused across turns.
+```
+trace → profile → human review → prune/compact decision
+```
 
-Related work:
-
-- **ByteDance Seed — _Scaling Long-Horizon LLM Agent via Context-Folding_**  
-  Studies context management for long-horizon agents through folding and summarizing intermediate sub-trajectories. This motivates `context-profiler`'s focus on turn-to-turn context diffs, retained observations, and compression/pruning evidence.
-- **SWE-agent — _Agent-Computer Interfaces Enable Automated Software Engineering_**  
-  Shows the importance of the agent-computer interface for software-engineering agents, motivating analysis of tool calls, terminal output, and artifact churn.
-- **WebArena — _A Realistic Web Environment for Building Autonomous Agents_**  
-  Demonstrates the value of realistic multi-step agent trajectories, motivating support for loop/transcript analysis rather than only single prompt snapshots.
+Built for both humans and agents:
+- **HTML reports** — interactive timeline, icicle, persistence heatmap, tools, diff, findings
+- **JSON contracts** — stable issue codes and evidence for automated agent workflows
+- **Trace-source agnostic** — same analysis across OpenAI, Anthropic, Langfuse, and public trajectory datasets
+
+## Report Views
+
+<table>
+  <tr>
+    <td width="50%">
+      <img src="assets/readme/semantic.png" alt="Icicle view with semantic color and diff mode" width="100%">
+      <br>
+      <strong>Icicle</strong>
+      <br>
+      Token distribution per request. Semantic color by role, diff mode for additions/removals.
+    </td>
+    <td width="50%">
+      <img src="assets/readme/persistence.png" alt="Persistence heatmap showing content blocks across requests" width="100%">
+      <br>
+      <strong>Persistence</strong>
+      <br>
+      Which content blocks survive across turns. Blue = token cost. Red = compact candidate.
+    </td>
+  </tr>
+  <tr>
+    <td width="50%">
+      <img src="assets/readme/tools.png" alt="Tools view with token table and invocation detail" width="100%">
+      <br>
+      <strong>Tools</strong>
+      <br>
+      Which tools dominate the context budget and their invocation details.
+    </td>
+    <td width="50%">
+      <img src="assets/readme/findings.png" alt="Findings drawer with grouped diagnosis issues" width="100%">
+      <br>
+      <strong>Findings</strong>
+      <br>
+      Issue codes with severity, evidence, and actionable recommendations.
+    </td>
+  </tr>
+</table>
+
+## Findings Across Public Datasets
+
+Profiled on real multi-turn agent trajectories from public benchmarks:
+
+| Dataset | Domain | Turns | Total Tokens | Redundancy | Top Issue | Carryover |
+|---------|--------|-------|-------------|------------|-----------|-----------|
+| [SWE-agent](https://huggingface.co/datasets/nebius/SWE-agent-trajectories) | Coding agent | 31 | 27.1K | 26.9% | `REPEATED_CONTENT_BLOCK` | 231K across 20 blocks |
+| [lmcache](https://huggingface.co/datasets/sammshen/lmcache-agentic-traces) | KV-cache traces | 35 | 36.5K | 1.4% | `REPEATED_CONTENT_BLOCK` | 403K across 20 blocks |
+| [OpenHands](https://huggingface.co/datasets/nvidia/SWE-Zero-openhands-trajectories) | Tool-heavy agent | 34 | 23.9K | 0.2% | `REPEATED_CONTENT_BLOCK` | 383K across 20 blocks |
+
+All examples are included in [`examples/`](examples/) with conversion scripts and pre-converted session files.
 
 ## Install
 
@@ -68,18 +98,17 @@ PYTHONPATH=src uv run context-profiler --version
 
 ## Quick Start
 
-Analyze a raw provider request:
+Analyze a multi-turn agent session (SWE-agent trajectory included):
 
 ```bash
-context-profiler analyze request.json --format auto
-context-profiler diagnose request.json --format auto --json
+context-profiler analyze examples/swe_agent/session.jsonl --format openai --html report.html
 ```
 
-Analyze a coding-agent transcript:
+Analyze a raw provider request:
 
 ```bash
-context-profiler diagnose cursor-transcript.jsonl --format cursor-jsonl --json
-context-profiler analyze claude-code-session.jsonl --format claude-code-jsonl --html report.html
+context-profiler analyze request.json --format auto
+context-profiler diagnose request.json --format auto --json
 ```
 
 Analyze a Langfuse export:
@@ -124,6 +153,32 @@ Generate an interactive report:
 context-profiler analyze session.jsonl --html report.html
 ```
 
+## CLI Output
+
+The terminal report gives a quick read on context budget, repeated content, and tool hotspots before you open the HTML report.
+
+```text
+╭──────────────────────────────────────────────────────────────────────────────╮
+│ context-profiler  |  mode: snapshot  |  source:                              │
+│ tests/fixtures/repeated_tool_calls.json                                      │
+╰──────────────────────────────────────────────────────────────────────────────╯
+
+⚠ Warnings
+  • Content duplication: 476 redundant tokens (60.2% of total)
+
+Token Distribution
+  Category                  Tokens    % of Total
+  Total Input                  791          100%
+    System Prompt               13          1.6%
+    Tool Definitions            83         10.5%
+    Messages (assistant)       609         77.0%
+    Messages (tool)             70          8.8%
+    Messages (user)             16          2.0%
+
+  Top Tools by Token Usage
+      generate_canvas_component    595    75.2%
+```
+
 ## Agent-Friendly CLI Harness
 
 `context-profiler` is strict about supported formats but helpful when input does not match. Agents can discover contracts and adapt unsupported traces without asking users to reshape data manually.
@@ -179,16 +234,6 @@ Use `context-profiler formats list --json` for the current machine-readable regi
 
 For `agent-transcript`, analysis is intentionally marked `partial`: hidden system prompts, rules, tool definitions, MCP schemas, and provider compaction may not be present.
 
-## HTML Report
-
-The HTML report is self-contained and keeps the existing profiler style:
-
-- **Icicle view**: hierarchical token breakdown, zoom, breadcrumb navigation.
-- **Context timeline**: growth over turns, with small markers for large additions/tool-input spikes.
-- **Diff mode**: unchanged / added / removed content between requests.
-- **Tools view**: per-tool token table and invocation details.
-- **Detail panel**: selected node or turn-level diff evidence.
-
 ## Example Diagnosis
 
 ```json
@@ -239,6 +284,19 @@ Recommended demo order:
 3. Langfuse trace export.
 4. Multi-turn academic trajectories such as `pagarsky/agent-trace`, `cx-cmu/agent_trajectories`, or SWE-agent traces.
 
+## Research Context
+
+`context-profiler` is motivated by recent work showing that long-horizon agents are constrained not only by model quality, but also by how their working context is retained, compressed, and reused across turns.
+
+Related work:
+
+- **ByteDance Seed — _Scaling Long-Horizon LLM Agent via Context-Folding_**  
+  Studies context management for long-horizon agents through folding and summarizing intermediate sub-trajectories. This motivates `context-profiler`'s focus on turn-to-turn context diffs, retained observations, and compression/pruning evidence.
+- **SWE-agent — _Agent-Computer Interfaces Enable Automated Software Engineering_**  
+  Shows the importance of the agent-computer interface for software-engineering agents, motivating analysis of tool calls, terminal output, and artifact churn.
+- **WebArena — _A Realistic Web Environment for Building Autonomous Agents_**  
+  Demonstrates the value of realistic multi-step agent trajectories, motivating support for loop/transcript analysis rather than only single prompt snapshots.
+
 ## Docs
 
 - [CLI harness design](docs/design/cli-harness.md)