Skip to content

Commit 62b0ede

Browse files
yanpgwangcursoragent
yanpgwang
and
cursoragent
committed
Promote internal notes to public docs
Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 934e380 commit 62b0ede

5 files changed

Lines changed: 240 additions & 1104 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,11 @@ Recommended demo order:
179179
3. Langfuse trace export.
180180
4. Multi-turn academic trajectories such as `pagarsky/agent-trace`, `cx-cmu/agent_trajectories`, or SWE-agent traces.
181181

182+
## Docs
183+
184+
- [CLI harness design](docs/design/cli-harness.md)
185+
- [Roadmap](docs/roadmap.md)
186+
182187
## What It Does Not Do
183188

184189
- It does not fetch traces from Langfuse, Hugging Face, Cursor, or Claude Code.

‎docs/design/cli-harness.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# CLI Harness Design
2+
3+
`context-profiler` is a trace-source agnostic context analysis harness for LLM agents.
4+
5+
The CLI is designed for two users at once:
6+
7+
- humans who want a readable report
8+
- coding agents that need discoverable commands, JSON contracts, and actionable validation errors
9+
10+
## Product Boundary
11+
12+
`context-profiler` analyzes traces. It does not fetch them.
13+
14+
Good upstream sources include:
15+
16+
- Langfuse CLI or API exports
17+
- OpenTelemetry / OpenInference span exports
18+
- raw OpenAI or Anthropic request logs
19+
- Cursor or Claude Code local transcripts
20+
- academic trajectory datasets
21+
22+
The agent or user brings the data. `context-profiler` validates, normalizes, diagnoses, and reports.
23+
24+
## Command Model
25+
26+
The CLI follows a discover/validate/analyze flow:
27+
28+
```bash
29+
context-profiler formats list --json
30+
context-profiler formats describe <format> --json
31+
32+
context-profiler schema trace --json
33+
context-profiler schema diagnosis --json
34+
35+
context-profiler validate <file|-> --format auto --json
36+
context-profiler normalize <file|-> --from auto --json
37+
38+
context-profiler diagnose <file|-> --format auto --json
39+
context-profiler analyze <file|-> --format auto --html report.html
40+
```
41+
42+
The style is inspired by mature agent-friendly CLIs:
43+
44+
- Langfuse CLI: schema-driven API access
45+
- kubectl: resource discovery and explainability
46+
- Terraform: validate/plan-style machine output
47+
- GitHub CLI: human commands plus machine-readable JSON
48+
- Repomix: AI-context oriented CLI and MCP distribution
49+
50+
## Input Kinds
51+
52+
Every format is classified by input kind and confidence.
53+
54+
| Input kind | Examples | Confidence |
55+
| --- | --- | --- |
56+
| `provider-request` | OpenAI, Anthropic | exact |
57+
| `observability-trace` | Langfuse, OTel/OpenInference | high |
58+
| `agent-transcript` | Cursor JSONL, Claude Code JSONL | partial |
59+
| `benchmark-trajectory` | agent-trace, agent_trajectories, SWE-agent | dataset-dependent |
60+
61+
`agent-transcript` inputs are useful for visible loop analysis but are not exact raw provider requests. They may omit hidden prompts, tool definitions, rules, MCP schemas, and provider-side compaction.
62+
63+
## Validation Contract
64+
65+
Validation should be strict but helpful.
66+
67+
Unknown input should not be silently guessed. Instead, `validate --json` returns:
68+
69+
- `valid: false`
70+
- stable error code
71+
- expected shape
72+
- `agent_action`
73+
- `next_steps`
74+
75+
This lets Cursor, Claude Code, or another agent inspect `schema trace --json`, adapt the input into `ContextTrace`, and retry without asking the user to manually reshape data.
76+
77+
## Diagnosis Contract
78+
79+
`diagnose --json` is the primary agent-facing output.
80+
81+
It returns:
82+
83+
- `analysis_scope`: input kind, confidence, limitations
84+
- `issues`: stable issue codes with evidence and recommendations
85+
- `diff_summary`: turn-to-turn added/removed/retained token facts
86+
- `diff_hints`: conservative hints such as large additions, high tool-use additions, and possible artifact churn
87+
88+
Issue codes should be evidence-first. Heuristic findings should use `possible_*` naming and include confidence.
89+
90+
## HTML Report
91+
92+
The HTML report is the human-facing view.
93+
94+
It should keep the existing visual language:
95+
96+
- dark, compact, monospace interface
97+
- timeline + icicle as the main layout
98+
- detail panel reused for selected nodes or turns
99+
- no heavy diagnosis dashboard unless the UI direction changes intentionally
100+
101+
Timeline markers should reflect evidence from `context_diff`, such as large additions and high tool-use turns. Clicking a marked turn should show the diff facts in the existing detail panel.
102+
103+
## Future Graph Layer
104+
105+
The long-term model is a Context Event Graph:
106+
107+
- `Run`
108+
- `Turn`
109+
- `Span`
110+
- `Message`
111+
- `Artifact`
112+
- `ContentBlock`
113+
- `Edge`
114+
115+
Initial graph-like evidence should remain deterministic:
116+
117+
- content hash
118+
- role and block type
119+
- tool name
120+
- tool call id
121+
- artifact key
122+
- request index
123+
124+
Avoid semantic stale-content claims until there is enough evidence. Prefer diff evidence plus conservative hints.

‎docs/roadmap.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Roadmap
2+
3+
This roadmap focuses on making `context-profiler` a top-tier open source context analysis harness for LLM agents.
4+
5+
## Current Focus
6+
7+
### 1. Context Diff Engine
8+
9+
Build reliable turn-to-turn evidence before making strong stale-content claims.
10+
11+
Current and near-term outputs:
12+
13+
- added tokens
14+
- removed tokens
15+
- retained tokens
16+
- top added blocks
17+
- top removed blocks
18+
- top tool additions
19+
- artifact keys
20+
- possible artifact churn
21+
22+
### 2. Tool Context Diagnosis
23+
24+
Make tool-driven context pressure obvious.
25+
26+
Current issue codes:
27+
28+
- `TOOL_USE_DOMINATES_CONTEXT`
29+
- `TOP_TOOL_CONTEXT_HOTSPOT`
30+
- `REPEATED_CONTENT_BLOCK`
31+
- `REPEATED_TOOL_INPUT`
32+
33+
Near-term improvements:
34+
35+
- distinguish tool input vs tool result pressure
36+
- suppress low-value repeated-field findings
37+
- improve artifact key extraction
38+
- add stronger evidence for repeated modification loops
39+
40+
### 3. Agent Skill Distribution
41+
42+
Ship a complete `analyze-agent-context` skill for Cursor, Claude Code, and Open Plugins compatible tools.
43+
44+
The skill should teach agents:
45+
46+
- do not fetch traces unless asked
47+
- validate any trace/loop/transcript before analysis
48+
- use `diagnose --json` for machine-readable findings
49+
- generate HTML only when useful for the user
50+
- explain confidence and limitations
51+
52+
## Planned Format Support
53+
54+
### Observability Traces
55+
56+
- OpenTelemetry / OpenInference spans
57+
- LangSmith run trees
58+
- richer Langfuse observation exports
59+
60+
### Coding Agent Trajectories
61+
62+
- SWE-agent `.traj` files
63+
- mini-SWE-agent output files
64+
- richer Claude Code subagent linkage
65+
66+
### Academic Multi-Turn Datasets
67+
68+
Prefer datasets with real turn-to-turn evolution:
69+
70+
- `pagarsky/agent-trace`
71+
- `cx-cmu/agent_trajectories`
72+
- SWE-agent trajectories
73+
74+
Toolathlon is not a priority first-class format because its multi-turn structure is less natural for context evolution analysis.
75+
76+
## Later
77+
78+
### MCP Server
79+
80+
Expose context-profiler as MCP tools after the CLI and skill workflow are stable.
81+
82+
Candidate tools:
83+
84+
- `validate_trace`
85+
- `diagnose_trace`
86+
- `generate_html_report`
87+
- `describe_format`
88+
- `get_schema`
89+
90+
### Context Event Graph
91+
92+
Evolve deterministic diff evidence into a graph model:
93+
94+
- repeated content edges
95+
- artifact modification chains
96+
- superseded context hints
97+
- orphaned context hints
98+
- subagent leakage hints
99+
100+
### Release Automation
101+
102+
Use GitHub Releases and PyPI Trusted Publishing for versioned releases.
103+
104+
Release checklist:
105+
106+
1. update `pyproject.toml`
107+
2. update `CHANGELOG.md`
108+
3. run smoke tests
109+
4. build package
110+
5. tag release
111+
6. publish through CI

0 commit comments

Comments
 (0)