Skip to content

Commit 3307151

Browse files
yanpgwangyanpgwang
committed
Initial release: context-profiler v0.1.0
Framework-agnostic profiler for LLM agent context windows. Made-with: Cursor
0 parents  commit 3307151

33 files changed

Lines changed: 5155 additions & 0 deletions

‎.github/workflows/ci.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
matrix:
14+
python-version: ["3.10", "3.11", "3.12", "3.13"]
15+
16+
steps:
17+
- uses: actions/checkout@v4
18+
19+
- name: Set up Python ${{ matrix.python-version }}
20+
uses: actions/setup-python@v5
21+
with:
22+
python-version: ${{ matrix.python-version }}
23+
24+
- name: Install dependencies
25+
run: |
26+
python -m pip install --upgrade pip
27+
pip install -e .
28+
pip install pytest
29+
30+
- name: Run tests
31+
run: pytest tests/ -v
32+
33+
- name: Verify CLI entry point
34+
run: context-profiler --help

‎.github/workflows/publish.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Publish to PyPI
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
permissions:
8+
id-token: write
9+
10+
jobs:
11+
build:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Set up Python
17+
uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.12"
20+
21+
- name: Install build tools
22+
run: python -m pip install --upgrade pip build
23+
24+
- name: Build package
25+
run: python -m build
26+
27+
- name: Upload dist artifacts
28+
uses: actions/upload-artifact@v4
29+
with:
30+
name: dist
31+
path: dist/
32+
33+
publish:
34+
needs: build
35+
runs-on: ubuntu-latest
36+
environment: pypi
37+
38+
steps:
39+
- name: Download dist artifacts
40+
uses: actions/download-artifact@v4
41+
with:
42+
name: dist
43+
path: dist/
44+
45+
- name: Publish to PyPI
46+
uses: pypa/gh-action-pypi-publish@release/v1

‎.gitignore‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.egg-info/
6+
*.egg
7+
dist/
8+
build/
9+
*.whl
10+
*.tar.gz
11+
12+
# Virtual environments
13+
.venv/
14+
venv/
15+
ENV/
16+
17+
# IDE
18+
.idea/
19+
.vscode/
20+
*.swp
21+
*.swo
22+
*~
23+
24+
# OS
25+
.DS_Store
26+
Thumbs.db
27+
28+
# Testing
29+
.pytest_cache/
30+
.coverage
31+
htmlcov/
32+
33+
# Generated reports (large files)
34+
report.html
35+
*.lhar
36+
37+
# Trace files
38+
trace-*.json
39+
40+
# Cursor / Claude
41+
.claude/
42+
.cursor/
43+
44+
# uv
45+
uv.lock
46+
47+
# Example generated files (keep raw data, ignore generated)
48+
examples/report.html
49+
examples/session.jsonl

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2025 Yanpeng Wang
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# context-profiler
2+
3+
[![PyPI version](https://img.shields.io/pypi/v/context-profiler.svg)](https://pypi.org/project/context-profiler/)
4+
[![Python](https://img.shields.io/pypi/pyversions/context-profiler.svg)](https://pypi.org/project/context-profiler/)
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
6+
7+
Framework-agnostic profiler for LLM agent context windows. Parses raw API request JSON and visualizes **where your tokens go** — no SDK instrumentation needed.
8+
9+
![Icicle view — token distribution breakdown](assets/demo-snapshot.gif)
10+
11+
## Why
12+
13+
LLM agent frameworks accumulate tool definitions, system prompts, and conversation history in the context window. You can't optimize what you can't see.
14+
15+
context-profiler gives you:
16+
17+
- **Token distribution** — breakdown by role, content type, and tool name
18+
- **Icicle visualization** — [speedscope](https://www.speedscope.app/)-style interactive view, zoom into any node
19+
- **Context growth timeline** — stacked area chart across a session, see the inflection point
20+
- **Diff visualization** — what's new vs. what's history vs. what got pruned, per request
21+
22+
![Session mode — timeline and diff](assets/demo-session.gif)
23+
24+
## Install
25+
26+
```bash
27+
pip install context-profiler
28+
```
29+
30+
Or install from source:
31+
32+
```bash
33+
git clone https://github.com/Turdot/context-profiler.git
34+
cd context-profiler
35+
pip install -e .
36+
```
37+
38+
## Quick Start
39+
40+
```bash
41+
# Analyze a single API request (snapshot mode)
42+
context-profiler analyze request.json
43+
44+
# Analyze context growth over multiple requests (session mode)
45+
context-profiler analyze session.jsonl
46+
context-profiler analyze requests_dir/
47+
48+
# Generate an interactive HTML report
49+
context-profiler analyze session.jsonl --html report.html
50+
51+
# Export JSON report
52+
context-profiler analyze request.json -o report.json
53+
54+
# Specify format explicitly
55+
context-profiler analyze request.json --format openai
56+
context-profiler analyze request.json --format anthropic
57+
58+
# Analyze Langfuse traces
59+
context-profiler analyze trace.json --format langfuse
60+
61+
# Multi-trace session (multiple Langfuse exports)
62+
context-profiler analyze trace1.json trace2.json trace3.json --html report.html
63+
```
64+
65+
## Supported Formats
66+
67+
Auto-detected from JSON structure:
68+
69+
| Format | Input | Mode |
70+
|--------|-------|------|
71+
| **OpenAI** | `{messages, tools}` | snapshot |
72+
| **Anthropic** | `{messages, tools}` with content blocks | snapshot |
73+
| **Langfuse trace** | `{observations: [{type: "GENERATION", ...}]}` | session |
74+
| **JSONL** | One request per line | session |
75+
| **Directory** | Folder of `.json` files | session |
76+
77+
## HTML Report Features
78+
79+
The HTML report is a self-contained file with no external dependencies:
80+
81+
- **Icicle view** — hierarchical token breakdown, click to zoom, breadcrumb navigation
82+
- **Tools view** — per-tool token table with stacked bars, sortable columns
83+
- **Timeline** — stacked area chart (system / tool defs / messages), click to select request
84+
- **Color modes** — Semantic (by role) or Diff (unchanged / added / removed)
85+
- **Role filters** — toggle visibility by role (system, user, assistant, tool)
86+
- **Detail panel** — content preview and JSON tree for any selected node
87+
88+
## CLI Output
89+
90+
```
91+
⚠ Warnings
92+
• Tool definitions consume 15.2K tokens (35.4% of total)
93+
94+
Token Distribution
95+
Category Tokens % of Total
96+
Total Input 42.9K 100%
97+
System Prompt 1.2K 2.8%
98+
Tool Definitions 15.2K 35.4%
99+
Messages (assistant) 8.4K 19.6%
100+
Messages (tool) 14.1K 32.9%
101+
Messages (user) 4.0K 9.3%
102+
103+
Top Tools by Token Usage
104+
Tool Tokens Calls
105+
playwright_with_chunk_browser_snap 12.4K 8
106+
filesystem-read_file 3.2K 5
107+
local-search_in_turn 1.1K 3
108+
```
109+
110+
## Examples
111+
112+
The `examples/` directory contains a complete demo using [Toolathlon](https://huggingface.co/datasets/hkust-nlp/Toolathlon-Trajectories) trajectories:
113+
114+
```bash
115+
cd examples/
116+
117+
# Convert Toolathlon data to context-profiler input
118+
python convert_toolathlon.py toolathlon_raw.json --mode snapshot -o snapshot.json
119+
python convert_toolathlon.py toolathlon_raw.json --mode session -o session.jsonl
120+
121+
# Analyze
122+
context-profiler analyze snapshot.json
123+
context-profiler analyze session.jsonl --html report.html
124+
```
125+
126+
See [`examples/README.md`](examples/README.md) for supported formats and conversion patterns.
127+
128+
## Acknowledgements
129+
130+
This project is inspired by and learned from:
131+
132+
- [context-lens](https://github.com/larsderidder/context-lens) — local proxy for capturing and visualizing LLM API calls
133+
- [ContextFlame](https://github.com/jcgs2503/contextflame) — flamegraph-based token profiling for Claude Code
134+
- [speedscope](https://www.speedscope.app/) — the icicle / flamegraph UI design is inspired by speedscope's interactive visualization
135+
136+
## License
137+
138+
[MIT](LICENSE)

‎assets/demo-session.gif‎

8.32 MB
Loading

‎assets/demo-snapshot.gif‎

5.94 MB
Loading

‎examples/README.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Examples
2+
3+
End-to-end demo: convert external data → profiler input → analysis report.
4+
5+
## Supported Input Formats
6+
7+
context-profiler natively accepts these formats (auto-detected):
8+
9+
| Format | Key Signature | Mode |
10+
|--------|--------------|------|
11+
| **OpenAI** | `{messages: [{role, content}], tools: [{type: "function", function: {...}}]}` | snapshot |
12+
| **Anthropic** | `{messages: [{role, content: [{type: "text"/"tool_use"/"tool_result", ...}]}], tools: [{name, input_schema}]}` | snapshot |
13+
| **Langfuse trace** | `{id, observations: [{type: "GENERATION", input: {messages, tools}}]}` | session |
14+
| **JSONL** (`.jsonl`) | One OpenAI/Anthropic request per line | session |
15+
| **Directory** | Folder of `.json` files, each one request | session |
16+
17+
## Demo: Toolathlon Trajectory → Report
18+
19+
[Toolathlon](https://huggingface.co/datasets/hkust-nlp/Toolathlon-Trajectories) is a public dataset with 5000+ agent execution trajectories from 17 LLMs. Its format is **not** directly compatible — this demo shows how to convert and analyze it.
20+
21+
### Raw data
22+
23+
`toolathlon_raw.json` — a GPT-5 agent executing a train ticket planning task:
24+
- 22 messages, 40 tool definitions, 11 LLM calls
25+
- ~80K total tokens across all calls
26+
27+
### Step 1: Convert to snapshot (single request)
28+
29+
```bash
30+
cd examples/
31+
32+
# Convert to a single OpenAI-format request (final context window)
33+
python convert_toolathlon.py toolathlon_raw.json --mode snapshot -o snapshot.json
34+
35+
# Analyze
36+
context-profiler analyze snapshot.json
37+
```
38+
39+
### Step 2: Convert to session (per-call snapshots)
40+
41+
```bash
42+
# Reconstruct each LLM call as a separate snapshot → JSONL
43+
python convert_toolathlon.py toolathlon_raw.json --mode session -o session.jsonl
44+
45+
# Analyze with session mode (shows context growth timeline)
46+
context-profiler analyze session.jsonl --html report.html
47+
```
48+
49+
### What you get
50+
51+
**Snapshot mode** — token distribution of the final (most bloated) API call:
52+
- How much of the context is tool definitions vs actual conversation
53+
- Which tools consume the most tokens
54+
55+
**Session mode** — context growth timeline across all 11 LLM calls:
56+
- How the message token count grows from call to call
57+
- Tool definitions stay constant while messages accumulate
58+
59+
## Adapting Other Formats
60+
61+
The same pattern works for any external format — write a small script that reshapes the data into OpenAI `{messages, tools}` format:
62+
63+
| Source Format | Key Difference | Conversion |
64+
|--------------|----------------|------------|
65+
| **ShareGPT** | `conversations`/`from`/`value` | Rename to `messages`/`role`/`content`, map `human`→`user`, `gpt`→`assistant` |
66+
| **LMSYS / WildChat** | `conversation` instead of `messages` | Rename field |
67+
| **Toolathlon** | Flat message array + stringified fields | Deserialize strings, split on assistant boundaries for session |
68+
| **LangSmith** | Hierarchical runs with `inputs`/`outputs` | Extract `run_type=llm` runs, use `inputs.messages` |

0 commit comments

Comments
 (0)