Skip to content

Commit 8310a93

Browse files
marcusschiessermarcusschiesser
committed
Add AgentPond skill and trace analysis references
1 parent e3355e9 commit 8310a93

6 files changed

Lines changed: 434 additions & 0 deletions

File tree

‎.changeset/agentpond-skill.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"agentpond": patch
3+
---
4+
5+
Add an AgentPond skill with CLI, DuckDB schema, ingestion, and trace-analysis guidance for coding agents.

‎skills/agentpond/SKILL.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
name: agentpond
3+
description: Work with AgentPond trace analytics. Use when needing to sync AgentPond object-store events into DuckDB, inspect traces, observations, sessions, or scores, create manual traces or scores, run local SQL analysis, or configure Langfuse-compatible SDK ingestion into AgentPond.
4+
allowed-tools:
5+
- Bash(npx agentpond --help *)
6+
- Bash(npx agentpond sync *)
7+
- Bash(npx agentpond traces create *)
8+
- Bash(npx agentpond traces list *)
9+
- Bash(npx agentpond traces get *)
10+
- Bash(npx agentpond observations list *)
11+
- Bash(npx agentpond sessions list *)
12+
- Bash(npx agentpond sessions get *)
13+
- Bash(npx agentpond scores create *)
14+
- Bash(npx agentpond scores list *)
15+
- Bash(npx agentpond sql *)
16+
---
17+
18+
# AgentPond
19+
20+
AgentPond is a data pond for AI agent traces with an agent-native CLI for local analytics. It accepts Langfuse-compatible ingestion, stores accepted events in S3-compatible object storage, and syncs those events into a local DuckDB cache for inspection.
21+
22+
## Core Principles
23+
24+
Follow these principles for AgentPond work:
25+
26+
1. **Sync before analysis**: Run `npx agentpond sync` before querying recent production data unless the user is intentionally inspecting an existing local cache.
27+
2. **Use supported CLI commands**: AgentPond exposes focused trace analytics commands and local SQL. Do not use Langfuse REST API discovery or arbitrary API-resource commands.
28+
3. **Use DuckDB for deeper analysis**: Prefer `npx agentpond sql "..."` when a question requires joins, aggregation, time filtering, raw event inspection, or cost analysis.
29+
4. **Keep credentials out of chat**: Ask the user to set environment variables or an `.env` file instead of pasting secrets into the conversation.
30+
31+
## Use Case References
32+
33+
- CLI usage and configuration: [references/cli.md](references/cli.md)
34+
- DuckDB tables and SQL examples: [references/duckdb-schema.md](references/duckdb-schema.md)
35+
- Langfuse SDK ingestion into AgentPond: [references/instrumentation.md](references/instrumentation.md)
36+
- Trace investigation and error analysis: [references/error-analysis.md](references/error-analysis.md)
37+
38+
## AgentPond CLI
39+
40+
Run AgentPond through `npx` unless the user has installed it globally:
41+
42+
```bash
43+
npx agentpond --help
44+
```
45+
46+
Configure the CLI with environment variables or an env file:
47+
48+
```bash
49+
export AGENTPOND_S3_ENDPOINT=http://localhost:9000
50+
export AGENTPOND_S3_BUCKET=agentpond
51+
export AWS_ACCESS_KEY_ID=minio
52+
export AWS_SECRET_ACCESS_KEY=minio123
53+
```
54+
55+
For a file-based setup:
56+
57+
```bash
58+
npx agentpond --env .env sync
59+
```
60+
61+
Common inspection flow:
62+
63+
```bash
64+
npx agentpond sync
65+
npx agentpond traces list --limit 25
66+
npx agentpond traces get <trace-id>
67+
npx agentpond observations list --traceId <trace-id>
68+
npx agentpond scores list --traceId <trace-id>
69+
```
70+
71+
Run SQL against the local DuckDB cache:
72+
73+
```bash
74+
npx agentpond sql "select id, name, session_id, total_cost from traces order by start_time desc limit 10"
75+
```
76+
77+
## Langfuse-Compatible Ingestion
78+
79+
AgentPond ingestion uses Langfuse-compatible endpoints. Applications can use normal Langfuse SDK configuration while pointing at an AgentPond ingestion service:
80+
81+
```bash
82+
export LANGFUSE_BASE_URL=http://localhost:3030
83+
export LANGFUSE_PUBLIC_KEY=pk-agentpond
84+
export LANGFUSE_SECRET_KEY=sk-agentpond
85+
```
86+
87+
These variables configure SDK ingestion. They are not credentials for using the AgentPond CLI to query Langfuse Cloud. The AgentPond CLI reads from object storage and the local DuckDB cache.

‎skills/agentpond/references/cli.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# AgentPond CLI
2+
3+
Use `npx agentpond` for AgentPond data access unless the user has installed the package globally.
4+
5+
## Configuration
6+
7+
AgentPond reads object storage and cache settings from environment variables:
8+
9+
```bash
10+
export AGENTPOND_PROJECT_ID=default-project
11+
export AGENTPOND_S3_BUCKET=agentpond
12+
export AGENTPOND_S3_PREFIX=
13+
export AGENTPOND_S3_ENDPOINT=http://localhost:9000
14+
export AGENTPOND_S3_FORCE_PATH_STYLE=true
15+
export AWS_ACCESS_KEY_ID=minio
16+
export AWS_SECRET_ACCESS_KEY=minio123
17+
export AWS_REGION=us-east-1
18+
export AGENTPOND_DB=~/.agentpond/cache.duckdb
19+
```
20+
21+
The CLI also accepts common settings as flags:
22+
23+
```bash
24+
npx agentpond --env .env sync
25+
npx agentpond --db ~/.agentpond/cache.duckdb sync
26+
npx agentpond --s3-bucket agentpond --s3-endpoint http://localhost:9000 sync
27+
```
28+
29+
## Sync
30+
31+
Sync reads accepted event manifests from object storage and projects them into the local DuckDB cache:
32+
33+
```bash
34+
npx agentpond sync
35+
npx agentpond sync --json
36+
```
37+
38+
Run sync before analysis when fresh data may exist in object storage.
39+
40+
## Traces
41+
42+
Create a manual trace:
43+
44+
```bash
45+
npx agentpond traces create \
46+
--name "manual trace" \
47+
--userId user_42 \
48+
--sessionId session_42 \
49+
--metadata '{"env":"local","feature":"checkout"}' \
50+
--input '{"question":"Why was my card declined?"}' \
51+
--output '{"answer":"The bank declined the authorization."}'
52+
```
53+
54+
List and read traces:
55+
56+
```bash
57+
npx agentpond traces list
58+
npx agentpond traces list --limit 25
59+
npx agentpond traces get <trace-id>
60+
```
61+
62+
## Observations
63+
64+
List observations for a trace:
65+
66+
```bash
67+
npx agentpond observations list --traceId <trace-id>
68+
```
69+
70+
## Sessions
71+
72+
List and read sessions:
73+
74+
```bash
75+
npx agentpond sessions list
76+
npx agentpond sessions get <session-id>
77+
```
78+
79+
## Scores
80+
81+
Create a score for a trace:
82+
83+
```bash
84+
npx agentpond scores create \
85+
--name quality \
86+
--value 0.9 \
87+
--traceId <trace-id>
88+
```
89+
90+
Create a human annotation score:
91+
92+
```bash
93+
npx agentpond scores create \
94+
--name reviewed \
95+
--value true \
96+
--traceId <trace-id> \
97+
--source ANNOTATION \
98+
--comment "Looks good"
99+
```
100+
101+
List scores for a trace or observation:
102+
103+
```bash
104+
npx agentpond scores list --traceId <trace-id>
105+
npx agentpond scores list --observationId <observation-id>
106+
```
107+
108+
## SQL
109+
110+
Run SQL against the local DuckDB cache:
111+
112+
```bash
113+
npx agentpond sql "select id, name, session_id from traces limit 10"
114+
npx agentpond sql "select * from scores where trace_id = '<trace-id>'"
115+
npx agentpond sql "select * from traces limit 10" --json
116+
```

‎skills/agentpond/references/duckdb-schema.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# DuckDB Schema
2+
3+
AgentPond stores raw accepted events and projected analysis tables in the configured DuckDB cache. Use `npx agentpond sync` before querying when object storage may contain new events.
4+
5+
## Queryable Structures
6+
7+
`events_raw` keeps every accepted event payload:
8+
9+
```text
10+
event_id TEXT PRIMARY KEY
11+
project_id TEXT
12+
manifest_key TEXT
13+
object_key TEXT
14+
event_type TEXT
15+
event_timestamp TIMESTAMP
16+
entity_id TEXT
17+
body_json TEXT
18+
event_json TEXT
19+
```
20+
21+
`traces` contains projected trace rows:
22+
23+
```text
24+
id TEXT PRIMARY KEY
25+
project_id TEXT
26+
name TEXT
27+
user_id TEXT
28+
session_id TEXT
29+
start_time TIMESTAMP
30+
end_time TIMESTAMP
31+
metadata_json TEXT
32+
input_json TEXT
33+
output_json TEXT
34+
total_cost DOUBLE
35+
updated_at TIMESTAMP
36+
```
37+
38+
`observations` contains projected span, generation, and observation rows:
39+
40+
```text
41+
id TEXT PRIMARY KEY
42+
project_id TEXT
43+
trace_id TEXT
44+
parent_observation_id TEXT
45+
type TEXT
46+
name TEXT
47+
start_time TIMESTAMP
48+
end_time TIMESTAMP
49+
metadata_json TEXT
50+
input_json TEXT
51+
output_json TEXT
52+
usage_details_json TEXT
53+
cost_details_json TEXT
54+
total_cost DOUBLE
55+
updated_at TIMESTAMP
56+
```
57+
58+
`scores` contains projected trace, observation, and session scores:
59+
60+
```text
61+
id TEXT PRIMARY KEY
62+
project_id TEXT
63+
trace_id TEXT
64+
observation_id TEXT
65+
session_id TEXT
66+
name TEXT
67+
value DOUBLE
68+
string_value TEXT
69+
data_type TEXT
70+
source TEXT
71+
comment TEXT
72+
metadata_json TEXT
73+
timestamp TIMESTAMP
74+
updated_at TIMESTAMP
75+
```
76+
77+
`sessions` is a view derived from traces with session IDs:
78+
79+
```text
80+
id TEXT
81+
project_id TEXT
82+
first_seen_at TIMESTAMP
83+
last_seen_at TIMESTAMP
84+
trace_count BIGINT
85+
```
86+
87+
## SQL Patterns
88+
89+
Recent high-cost traces:
90+
91+
```bash
92+
npx agentpond sql "select id, name, session_id, total_cost from traces order by total_cost desc nulls last limit 20"
93+
```
94+
95+
Observation timeline for one trace:
96+
97+
```bash
98+
npx agentpond sql "select id, parent_observation_id, type, name, start_time, end_time, total_cost from observations where trace_id = '<trace-id>' order by start_time asc"
99+
```
100+
101+
Scores attached to a trace:
102+
103+
```bash
104+
npx agentpond sql "select name, value, string_value, data_type, source, comment, timestamp from scores where trace_id = '<trace-id>' order by timestamp desc"
105+
```
106+
107+
Sessions with repeated trace activity:
108+
109+
```bash
110+
npx agentpond sql "select id, trace_count, first_seen_at, last_seen_at from sessions order by trace_count desc, last_seen_at desc limit 20"
111+
```
112+
113+
Raw event inspection:
114+
115+
```bash
116+
npx agentpond sql "select event_type, event_timestamp, entity_id, body_json from events_raw where entity_id = '<entity-id>' order by event_timestamp asc"
117+
```
118+
119+
JSON columns are stored as text. Use DuckDB JSON functions when a query needs fields inside `metadata_json`, `input_json`, `output_json`, `usage_details_json`, `cost_details_json`, `body_json`, or `event_json`.

0 commit comments

Comments
 (0)