Add export v1 contracts and session summary export by wesm · Pull Request #991 · kenn-io/agentsview

wesm · 2026-07-04T03:32:48Z

This PR turns report/export JSON into explicit v1 contracts for programmatic consumers. It adds shared schema, pricing provenance, and project identity metadata to usage daily and activity report outputs, and introduces a daemonless agentsview export sessions summary export for headless analytics. The session export is content-free, supports JSON/NDJSON, and includes per-session usage, model, cost, project, worktree, branch, machine, timestamp, and classification metadata without transcript content.

Pricing provenance is centralized under internal/export: reports use a resolver-derived block with source/table metadata, RFC 8785-style digest, fallback indicators, cost_source, and a bounded per-model effective rates map. Source-reported costs are marked so consumers know when token-times-rate recomputation is not expected, and reasoning tokens are handled as output-rate billing breakdowns.

Project identity now persists raw observations at sync/import time and recomputes stable identities at export time. Remote-backed identities use normalized network remotes with sha256: keys; path-backed fallbacks remain explicit and machine-local. The identity store is preserved through resync and mirrored through PostgreSQL/DuckDB so CLI and HTTP exports stay aligned across backends.

The new session-summary export adds stable watermark/keyset pagination, cursor-reset signaling, --all, NDJSON meta rows, root/child and automation filtering, and shared pricing/project metadata. Existing usage/activity payloads stay additive: metadata lands as sibling blocks, and daily breakdown arrays are pinned as arrays rather than omitted.

Docs now describe the v1 contract rules, pricing digest input, project identity derivation, cursor behavior, session-export limits, and default exclusion caveats. Golden fixtures pin usage daily, usage daily with breakdowns, activity report, and session export JSON/NDJSON shapes. Stale docs/superpowers design notes were removed, and the shared contract package was renamed from internal/exportcontracts to internal/export.

Reviewers should focus on:

shared DTO/resolver code in internal/export
project identity capture, fallback, resync preservation, and mirror-backend persistence
session summary export query/cursor behavior in internal/db/session_export.go and cmd/agentsview/export.go
pricing provenance coupling across SQLite, PostgreSQL, and DuckDB usage/activity paths

The main tradeoff is landing the related export-contract issues together so field names and semantics stay shared across surfaces. This intentionally does not add redaction flags or per-row pricing provenance: raw project paths/remotes are emitted by default, and pricing provenance remains report-level with a bounded per-model map.

Programmatic consumers need stable JSON contracts for report and session-summary exports, including reproducible pricing metadata and project identity that can survive cross-machine aggregation. This commit publishes the final branch state without the private local-path examples that existed only in intermediate local commits. The export surfaces stay additive while gaining shared schema metadata, pricing provenance, persisted project identity, and a content-free daemonless session summary export for headless analytics.

V1 report metadata should distinguish an empty projects map from legacy absence. Usage summary and activity responses now keep the projects field present, with regression coverage at the DB, activity, service, and HTTP response layers.

The v1 activity report golden only needs enough rows to pin the contract surface. The prior full-day default generated hundreds of empty five-minute buckets, making review noisy without adding distinct schema coverage. Use a narrow custom window around the seeded sessions so the fixture still carries pricing, project identity, partial-range metadata, nonzero buckets, summaries, sessions, and intervals while staying small enough to read.

roborev-ci · 2026-07-04T03:58:32Z

roborev: Combined Review (`c96e40e`)

High: internal/postgres/push.go:509 persists raw git remotes into shared PostgreSQL. Since GitRemote is read unchanged from .git/config at internal/sync/engine.go:6628-6630, credential-bearing remotes like https://user:token@github.com/org/repo.git can leak tokens to DB readers/operators. Sanitize before storage or persist only the normalized remote/key, and add a migration to scrub existing rows.

Medium: cmd/agentsview/export.go:196 opens SQLite via openDB, bypassing the write-owner lock/live daemon coordination. The export path can still write through GetOrCreateDatabaseID, so running agentsview export sessions beside a writable server may create an uncoordinated second writer. Route export through the locked write path when metadata may be created, or make export strictly read-only by reading an existing DB ID only.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 17m41s

V1 export metadata is now consumed as join and provenance data, so it must not leak credential-bearing remotes or mutate archives from daemonless read commands. Persist sanitized project remotes and scrub existing observations while preserving remote-derived project keys.\n\nSession-summary export now requires an existing archive database ID and uses the read-only open path, avoiding an uncoordinated SQLite writer beside the daemon.\n\nPostgreSQL and DuckDB now treat explicit pricing rows as the effective pricing table, matching SQLite provenance digests while retaining fallback pricing for empty fresh mirrors. The docs also call out that the new JSON contracts aim for compatibility but may still settle.

roborev-ci · 2026-07-04T11:30:37Z

roborev: Combined Review (`367a7ea`)

High risk overall: reviewers found one production-blocking export issue plus two medium integrity/ordering issues.

High

cmd/agentsview/export.go:203: export sessions requires an existing database_id, but no production path creates one. GetOrCreateDatabaseID is only used in tests, so real archives can keep failing with “database id missing” even after restarting serve.
- Fix: Initialize the database ID from a writable startup/open path, such as after db.Open/migration during serve, before read-only export relies on it.

Medium

internal/db/session_export.go:424: Session export orders and paginates by raw RFC3339Nano text timestamps. Whole-second values like ...00Z sort after fractional values like ...00.999Z lexicographically, so latest-activity ordering and cursor boundaries can be wrong within the same second.
- Fix: Use a normalized sortable timestamp key for watermark, comparisons, prefix fingerprints, and ORDER BY, such as fixed-width fractional UTC text or parsed epoch/nanosecond components.
internal/db/session_export.go:653: Session export cursors are HMAC-signed with a public constant instead of the archive cursor secret, so callers can forge valid cursors and bypass the filter/database/prefix integrity checks that the signature is meant to enforce.
- Fix: Sign and verify export cursors with db.cursorSecret, optionally domain-separated for session export.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 20m12s

Session-summary export is intentionally read-only, so fresh archives need their database identity created by the normal writable startup path rather than by the export command. This keeps daemonless export from opening a second writer while letting archives initialized by serve or sync export immediately.\n\nDuckDB mirrors also need a repair for rows written before remote sanitization. The mirror schema migration now rewrites credential-bearing project identity remotes to their sanitized storage form and records a repair marker so compatibility checks can detect mirrors that still need the scrub.

roborev-ci · 2026-07-04T12:13:50Z

roborev: Combined Review (`ec435ca`)

Summary verdict: one medium-severity issue should be fixed before merge.

Medium

internal/db/session_export.go:798: last_activity_at is derived with MAX() over raw timestamp text and then reused for ordering, watermarks, and cursor comparisons. SQLite sorts RFC3339/RFC3339Nano strings lexically, so timestamps like ...00.123Z can sort before ...00Z, causing incorrect export ordering, paging, or reported activity when sub-second timestamps are present.

Fix: Use a normalized sortable timestamp value for max/order/cursor predicates, and add a regression test with mixed Z and fractional timestamps.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 21m33s

The v1 export surfaces are now join/provenance contracts, so edge cases that previously looked harmless can silently change consumer semantics. This tightens cursor snapshots, project identity normalization, backend mirror parity, pricing provenance, and JSON compatibility so empty, stale, and cross-backend cases remain distinguishable. The sync path also caches project identity discovery and makes identity write failures non-fatal in batch processing, avoiding per-append filesystem work while preserving session ingestion.

Stored project identity keys need to be reproducible across live and persisted observations. Windows drive-letter paths also need the same symlink resolution as live paths, and DuckDB mirror fallback must treat slash-form stored file paths consistently on Windows.

roborev-ci · 2026-07-04T13:51:19Z

roborev: Combined Review (`757b97a`)

Medium finding: SQLite export pricing can silently fall back to zero cost.

Medium: internal/db/usage.go:1536
SQLite loadPricingMap returns an empty effective pricing table when model_pricing has no rows, while PostgreSQL and DuckDB fall back to the embedded catalog. The new export sessions path calls this directly from a read-only DB, so a fresh sync-only archive can export tokenized sessions as unpriced with zero cost.
Fix: Populate embedded fallback pricing when no non-metadata pricing rows are loaded, then overlay custom pricing.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 16m34s

Live and persisted project identity observations must derive the same root_path-backed key. Windows drive-letter paths now use the same symlink resolution path in both normalizers, so a stored observation cannot diverge from the live observation that created it.

roborev-ci · 2026-07-04T14:17:07Z

roborev: Combined Review (`8db71c1`)

Medium: Filtered export cursors cannot be resumed from the CLI.

cmd/agentsview/export.go:247
--cursor rejects all query-affecting flags, but collectExportSessionPages still passes the current/default filter into ExportSessionSummaries, which compares it to the cursor’s embedded filter. A cursor minted from a filtered first page, such as --project alpha --limit 1, cannot be resumed because the second command is forbidden from repeating --project and then fails with a filter conflict.

Fix: When --cursor is present, hydrate the export filter from the cursor or make the DB export path use the cursor’s embedded filter for resumed pages. Add a CLI test for paginating a filtered export.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 26m34s

CLI resume commands cannot repeat query filters because the cursor is meant to carry the query contract. A cursor minted from a filtered first page therefore needs to hydrate the resumed export from the embedded filter instead of comparing it against the default empty CLI filter.\n\nDirect database callers still get the existing conflict check unless they explicitly opt into cursor-owned filtering, so programmatic misuse remains visible while the CLI path can paginate filtered exports.

roborev-ci · 2026-07-04T19:46:40Z

roborev: Combined Review (`a6b73c3`)

Summary verdict: One medium-severity issue remains; no high or critical findings were reported.

Medium

internal/db/usage.go:1761, internal/postgres/usage.go:1242, internal/duckdb/analytics_usage.go:3618
Cursor usage rows are added to the Projects metadata map with an empty project key because cursor rows set project = '' and aggregation records every row project unconditionally. Usage summaries can expose a bogus "" project identity whenever Cursor usage is present.

Fix: only add real session project labels to the project set, for example by skipping usage_source == "cursor" or consistently skipping empty project labels across SQLite, PostgreSQL, and DuckDB. Add parity tests for cursor-only usage metadata.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 11m38s

Cursor billing rows intentionally do not have a session project label, but the usage export metadata treated every aggregate row project as a join key. That leaked a synthetic empty-string project identity into v1 payloads whenever a report contained only Cursor usage. The totals and breakdown aggregation still preserve those Cursor rows; only the report-level project identity map now skips non-label projects so legacy absence and real project identities stay distinguishable across SQLite, PostgreSQL, and DuckDB. VALID (fixed): #1 -- Cursor-only usage no longer emits a "" project identity in usage report metadata.

roborev-ci · 2026-07-04T20:31:06Z

roborev: Combined Review (`f2150c9`)

Medium finding only:

internal/db/usage.go:349: Message-level reasoning_tokens are still projected as 0 even though the new pricing paths now charge reasoning tokens at the output rate. Sessions whose message token_usage includes reasoning_tokens will undercount usage and cost in reports/exports; the same pattern appears in the PostgreSQL and DuckDB usage paths.
- Fix: Extract reasoning_tokens from message token usage JSON in SQLite, PostgreSQL, and DuckDB, thread it through the existing usage rows, and add a parity test with message-only reasoning tokens.

Reviewers: 2 done | Synthesis: codex, 5s | Total: 28m28s

Message token_usage can carry reasoning_tokens, and pricing now charges those tokens at the output rate. Treating message rows as zero made daily usage, session usage, activity report cost, and session-summary export counters undercount message-only reasoning even though usage-event rows were already handled.\n\nKeep SQLite and PostgreSQL cost paths on their tolerant token JSON parsers while projecting message reasoning in the row templates, and add parity coverage across SQLite, PostgreSQL, and DuckDB.

Message token_usage remains raw provider JSON, so PostgreSQL usage SQL must not cast fields before the tolerant Go parser can clamp them. Leave message reasoning projected as zero in SQL and rely on the existing parsed message path for cost calculations, matching the SQLite behavior and avoiding query failures for out-of-range provider values.\n\nVALID (fixed): #27379 -- PostgreSQL message reasoning tokens no longer cast raw token_usage in SQL; pgtest now covers an out-of-range reasoning value.

roborev-ci · 2026-07-04T22:07:09Z

roborev: Combined Review (`ce4158e`)

Overall verdict: one medium-severity issue should be addressed before merge.

Medium

internal/db/session_export.go:23
Export cursors are HMAC-signed with a public constant, allowing consumers to forge cursors. Since the CLI rejects filter flags when --cursor is present, then trusts the cursor’s embedded filters and watermark, a less-privileged consumer allowed to resume a scoped export could modify those values, recompute the HMAC, and retrieve session summary metadata outside the original scope.

Suggested remediation: sign export cursors with a per-install or per-database secret, such as the existing configured cursor_secret or a random secret stored in archive metadata, and reject cursors signed with the old public key.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 23m31s

roborev-ci · 2026-07-04T22:50:03Z

roborev: Combined Review (`73cf809`)

Public docs build has one medium-risk regression; no high or critical findings were reported.

Medium

docs/zensical-docs.sh:64: The public docs build no longer excludes docs/superpowers, while this change adds internal plan/spec markdown under that tree. Those files can now be copied into the public docs build, risking publication of implementation notes or strict docs build failures for pages not intended for the public site.
- Fix: Restore --exclude './superpowers' in the tar command, or move intentionally public content into the normal docs tree while keeping internal plan/spec files excluded.

Reviewers: 2 done | Synthesis: codex, 6s | Total: 19m2s

roborev-ci · 2026-07-04T23:12:13Z

roborev: Combined Review (`4e40823`)

Summary verdict: One medium-severity issue remains; no high or critical findings were reported.

Medium

cmd/agentsview/export.go:199
export sessions opens the archive read-only and never seeds or applies fallback pricing before calling ExportSessionSummaries. A fresh archive created by direct sync, before serve or usage commands seed model_pricing, can export fallback-priced models as unpriced with zero cost and incomplete pricing provenance.

Fix: Ensure export installs effective fallback plus custom pricing for read-only archives when the pricing table is empty, without overriding existing fetched rows.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 21m33s

roborev-ci · 2026-07-05T01:09:51Z

roborev: Combined Review (`bda7576`)

Reviewed the changed export, cursor-signing, project identity, pricing metadata, and SQLite/PostgreSQL/DuckDB query paths for security-impacting regressions.

No issues found.

Reviewers: 2 total (1 done, 1 skipped) | Synthesis: codex | Total: 50m35s

…vice layer) Waypoint C of the staged upstream catch-up (specs/merge-upstream-waypoints.md). Merges u/main up to 725a3d0 (126 commits: export v1 foundation, the export.PricingResolver cost model, a service layer for usage summaries, the kit-ui component library + TopBar/theme, i18n, pairwise usage comparison, new Activity/RecentEdits nav) into our branch. Pricing (thread the fork's 1h cache-write rate through export.PricingResolver without touching the frozen export v1 wire contract): - Add an internal, non-serialized CacheWrite1hPerMTok to export.ModelRates (no canonical_json key, no EffectiveModelRate field -> digest/wire stay byte-identical), populated from db.ModelPricing.CacheCreation1hPerMTok in loadPricingMap/modelPricingRates with the defensive 5m fallback. - Restore the 5m/1h split in dailyUsageAmounts and sessionRowCost via a new parseEphemeralCacheSplit helper (upstream's rewritten token parser skips the nested cache_creation object), computing cost inline so 1h bills at its rate. - Drop the now-orphaned db.modelRates struct; the export resolver is the single cost source. Usage summary: re-home the snapshot exporter's BuildUsageSummary onto the new service layer (service.BuildUsageSummaryResult + usageSummaryResponseFromService), since the fold*/computeCacheStats helpers moved into internal/service. Frontend: adopt kit-ui's TopBar header, filtering the nav `tabs` to hide curation routes in snapshot mode (replaces the old {#if !SNAPSHOT} nav); adopt kit-ui theme getters/setters in ui.svelte.ts; keep the fork's 24-color golden-angle + org-theming projectColor over kit-ui's hashColor; coexist the longestPrompts panel with the new pairwise-comparison panel in the usage store (Endpoint now spans summary|comparison|pairwise|topSessions|longestPrompts). CLI: `usage projects` and `usage longest-prompts` adopt the paired --format/--json convention upstream now enforces. Deferred (unchanged from B): 1h-cache parity for the PostgreSQL and DuckDB cost paths; SetSessionPrivate stays read-only on the PG store. Green: go build/vet/test (except the snapshot test needing a git `origin` remote), svelte-check, 1770 vitest tests, make build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

wesm added 3 commits July 3, 2026 22:31

fix: emit empty export project maps

db5ed25

V1 report metadata should distinguish an empty projects map from legacy absence. Usage summary and activity responses now keep the projects field present, with regression coverage at the DB, activity, service, and HTTP response layers.

wesm added 2 commits July 4, 2026 08:28

wesm added 2 commits July 4, 2026 16:36

fix: sign session export cursors with install secret

73cf809

fix: keep superpowers tree out of public docs build

4e40823

fix: fall back to embedded pricing in session export

bda7576

wesm merged commit 725a3d0 into main Jul 5, 2026
24 checks passed

wesm deleted the dust-promotion-pr branch July 5, 2026 01:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add export v1 contracts and session summary export - #991

Add export v1 contracts and session summary export#991
wesm merged 15 commits into
mainfrom
dust-promotion-pr

wesm commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wesm commented Jul 4, 2026

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (c96e40e)

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (367a7ea)

High

Medium

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (ec435ca)

Medium

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (757b97a)

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (8db71c1)

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (a6b73c3)

Medium

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (f2150c9)

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (ce4158e)

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (73cf809)

Medium

Uh oh!

roborev-ci Bot commented Jul 4, 2026

roborev: Combined Review (4e40823)

Medium

Uh oh!

roborev-ci Bot commented Jul 5, 2026

roborev: Combined Review (bda7576)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

roborev: Combined Review (`c96e40e`)

roborev: Combined Review (`367a7ea`)

roborev: Combined Review (`ec435ca`)

roborev: Combined Review (`757b97a`)

roborev: Combined Review (`8db71c1`)

roborev: Combined Review (`a6b73c3`)

roborev: Combined Review (`f2150c9`)

roborev: Combined Review (`ce4158e`)

roborev: Combined Review (`73cf809`)

roborev: Combined Review (`4e40823`)

roborev: Combined Review (`bda7576`)