Skip to content

Commit e8d34ae

Browse files
committed
feat(cli): improve local archive ergonomics
1 parent 2f444ee commit e8d34ae

19 files changed

Lines changed: 1505 additions & 55 deletions

File tree

README.md

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Try the archive path with synthetic fixtures from a source checkout:
4141

4242
```bash
4343
fincrawl sync --fixture testdata/synthetic
44-
fincrawl search "billing refund" --fields provider_id,subject,updated_at
44+
fincrawl search "billing refund" --fields provider_id,subject,score,updated_at
4545
```
4646

4747
For live Intercom access, use a tenant-authorized read-only token from your
@@ -51,7 +51,8 @@ environment or ignored local env file:
5151
fincrawl sync --entities
5252
fincrawl sync --updated-since 2h --limit 50
5353
fincrawl sync --updated-since 180d --updated-before 90d --limit 0
54-
fincrawl search "login code expired" --fields provider_id,subject,updated_at
54+
fincrawl search "login code expired" --fields provider_id,subject,score,updated_at
55+
fincrawl show <intercom-conversation-id> --fields provider_id,subject,tags,snippet
5556
```
5657

5758
Use exact hydration when you already know a conversation ID:
@@ -81,6 +82,12 @@ FINCRAWL_AGE_IDENTITY=<age-identity> \
8182
Plaintext archive output is local scratch data only. Tenant encrypted snapshots
8283
still belong in tenant-controlled private storage, not in this repository.
8384

85+
Verify a tenant-controlled encrypted store before importing from it:
86+
87+
```bash
88+
fincrawl store verify <tenant-store-root>
89+
```
90+
8491
## Commands
8592

8693
The CLI defaults to JSON output for agents and automation:
@@ -101,11 +108,13 @@ Common flows:
101108
| Sync a recent window | `fincrawl sync --updated-since 2h --limit 50` |
102109
| Backfill a bounded historical window | `fincrawl sync --updated-since 180d --updated-before 90d --limit 0` |
103110
| Hydrate one conversation | `fincrawl sync --conversation <id>` |
104-
| Search local archive | `fincrawl search "<query>" --fields provider_id,subject,updated_at` |
111+
| Search local archive | `fincrawl search "<query>" --fields provider_id,subject,score,updated_at` |
112+
| Show one conversation | `fincrawl show <id> --fields provider_id,subject,tags,snippet` |
105113
| Filter search results | `fincrawl search "<query>" --state open --tag billing` |
106114
| Find Fin-status matches | `fincrawl search "<query>" --fin-status resolved` |
107115
| Export encrypted snapshot | `fincrawl publish --recipient <recipient> --out snapshots/local.jsonl.zst.age` |
108116
| Import encrypted snapshot | `fincrawl import --identity <identity> --in snapshots/local.jsonl.zst.age` |
117+
| Verify an encrypted tenant store | `fincrawl store verify <path>` |
109118
| Check repo guardrails | `fincrawl guard --json` |
110119

111120
## Docs

docs/architecture.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,13 @@ fincrawl sync --updated-since 180d --updated-before 90d --limit 0
2424
fincrawl sync --resume
2525
fincrawl sync --conversation <id>
2626
fincrawl search "billing refund" --json
27+
fincrawl show <id> --fields provider_id,subject,tags,snippet
2728
fincrawl search "billing refund" --state open --tag billing
2829
fincrawl search "login" --fin-status resolved
2930
fincrawl archive --fixture testdata/synthetic --recipient <age-recipient> --out tmp/archive.jsonl.zst.age
3031
fincrawl publish --recipient <age-recipient> --out snapshots/local.jsonl.zst.age
3132
fincrawl import --identity <age-identity> --in snapshots/local.jsonl.zst.age
33+
fincrawl store verify <tenant-store-root>
3234
fincrawl guard --json
3335
```
3436

@@ -46,6 +48,7 @@ internal/control/ machine-readable command descriptions and safe examples
4648
internal/intercom/ provider API client boundary
4749
internal/syncer/ fixture, entity, exact, and tail sync orchestration
4850
internal/store/ SQLite schema, migrations, export, search, sync state
51+
internal/tenantstore/ generic encrypted tenant-store manifest checks
4952
internal/archive/ canonical JSONL, zstd compression, age encryption
5053
internal/guard/ preflight repository leak checks
5154
internal/lock/ local write lock
@@ -72,7 +75,9 @@ and small:
7275
- `raw_blobs`
7376
- `conversation_fts`
7477

75-
Search uses FTS with sanitized queries and a fallback path where needed. Raw
78+
Search uses FTS with sanitized queries, result scores, compact field masks, and
79+
a fallback path where needed. `show` resolves either local IDs or provider IDs;
80+
conversation parts are opt-in and text snippets are sanitized before output. Raw
7681
provider JSON is retained locally for replay and migration debugging, but it is
7782
tenant data and must not be committed to this repo.
7883

@@ -99,12 +104,18 @@ age encryption. Supported artifact shape:
99104

100105
```text
101106
*.jsonl.zst.age
107+
*.tar.zst.age
102108
```
103109

104110
Plaintext archives, databases, logs, snapshots, screenshots, reports, and
105111
transcripts are ignored and blocked by `fincrawl guard`. Encrypted tenant
106112
snapshots are still tenant data and belong outside this repo.
107113

114+
Tenant stores can be checked locally with `fincrawl store verify <path>`. The
115+
verifier reads `manifest.json`, requires manifest snapshots to point at existing
116+
compressed age-encrypted artifacts, and rejects plaintext archives, SQLite
117+
stores, runtime state, logs, reports, screenshots, and transcripts.
118+
108119
## Agent Use
109120

110121
Agents should load [the repo skill](../skills/fincrawl/SKILL.md) before using a

docs/roadmap.md

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,27 +12,32 @@ this repo.
1212
- Offline `doctor`, `metadata`, `status`, `describe`, and `guard` commands.
1313
- Synthetic fixture sync for deterministic tests.
1414
- SQLite archive with FTS-backed search.
15-
- Search filters for state, tag, and Intercom-exposed Fin status.
15+
- Search ranking, compact field masks, NDJSON streams, and filters for state,
16+
tag, and Intercom-exposed Fin status.
17+
- `show` for one local conversation by local ID or provider ID, with sanitized
18+
snippets and opt-in sanitized parts.
1619
- Read-only Intercom entity sync for admins, teams, tags, and capped contacts
1720
when scopes allow.
1821
- Exact conversation hydration and bounded updated-since / updated-before tail
1922
sync.
2023
- Resumable sync state with privacy-safe status output.
2124
- Canonical JSONL export from fixtures or local SQLite.
2225
- zstd + age encrypted `archive`, `publish`, and `import` flows.
26+
- Generic `store verify` checks for tenant-controlled encrypted snapshot
27+
manifests.
2328
- Repository guardrails for plaintext archives, generated artifacts, secret
2429
patterns, provider URLs, and transcript-like files.
2530
- Agent-facing skill guidance under `skills/fincrawl/`.
2631
- CI/release automation for `0.0.x` bootstrap releases with protected `main`.
2732

2833
## Next
2934

30-
- Tighten local search result ranking and result fields from stored entity data.
31-
- Improve import/search ergonomics for read-only subscribers.
32-
- Add a small, generic tenant-store wrapper contract without committing tenant
33-
config or artifacts here.
34-
- Add focused tests for more provider pagination, rate-limit, and resume edge
35-
cases.
35+
- Add a read-only subscriber flow that can pull/import a local tenant store
36+
path without adding remote push or schedule mechanics.
37+
- Broaden exact hydration/search ergonomics around known provider URLs while
38+
keeping provider IDs path-safe.
39+
- Add more sync torture coverage around interrupted multi-page windows and
40+
repeated transient failures.
3641

3742
## Later
3843

docs/tenant-data-boundary.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,12 @@ Plaintext archive outputs are local-only scratch data and must be ignored or
8181
blocked by preflight checks. Tenant encrypted artifacts are still tenant data and
8282
belong in tenant-controlled private storage, not in `uinaf/fincrawl`.
8383

84+
Use `fincrawl store verify <tenant-store-root>` on tenant-controlled stores
85+
before importing from them. The verifier expects `manifest.json` to reference
86+
existing compressed age-encrypted snapshots with relative paths and rejects
87+
plaintext archives, local databases, runtime state, logs, reports, screenshots,
88+
and transcripts.
89+
8490
Encryption recipients may be native `age1...` recipients or SSH public keys
8591
accepted by age. Private age identities and private SSH keys must stay outside
8692
the repo. Import dry-runs still need a private decrypt identity because they

internal/cli/agent_dx.go

Lines changed: 118 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -165,9 +165,29 @@ func describeCommands(command string) (cliSchema, error) {
165165
"fincrawl search \"billing refund\" --limit 10",
166166
"fincrawl search \"billing refund\" --state open --tag billing",
167167
"fincrawl search \"login\" --fin-status resolved",
168-
"fincrawl search \"billing refund\" --fields provider_id,subject,updated_at --ndjson",
168+
"fincrawl search \"billing refund\" --fields provider_id,subject,score,updated_at --ndjson",
169169
},
170-
Notes: []string{"Allowed fields: id, provider_id, subject, state, assignee, rating, fin_status, participants, tags, updated_at, snippet."},
170+
Notes: []string{"Allowed fields: id, provider_id, subject, state, assignee, rating, fin_status, participants, tags, updated_at, snippet, score."},
171+
},
172+
"show": {
173+
Name: "show",
174+
Summary: "Show one local conversation by local ID or provider ID.",
175+
JSON: true,
176+
Args: []paramSchema{
177+
{Name: "id", Type: "conversation-id|provider-id", Required: true, Help: "Conversation local ID or provider ID."},
178+
},
179+
Flags: []paramSchema{
180+
{Name: "fields", Type: "field-list", Help: "Comma-separated fields to include in output."},
181+
{Name: "parts", Type: "bool", Help: "Include sanitized conversation parts."},
182+
{Name: "part-limit", Type: "int", Default: "20", Help: "Maximum parts when --parts is set."},
183+
{Name: "json", Type: "bool", Default: "true", Help: "Print JSON output."},
184+
},
185+
Examples: []string{
186+
"fincrawl show <provider-conversation-id>",
187+
"fincrawl show <provider-conversation-id> --fields provider_id,subject,tags,snippet",
188+
"fincrawl show <provider-conversation-id> --parts --part-limit 5",
189+
},
190+
Notes: []string{"Allowed fields: id, provider_id, subject, state, assignee, rating, fin_status, participants, tags, created_at, updated_at, snippet, parts.", "Parts are opt-in and sanitized before output."},
171191
},
172192
"archive": {
173193
Name: "archive",
@@ -218,6 +238,22 @@ func describeCommands(command string) (cliSchema, error) {
218238
"Input paths must be relative, stay under the current working directory, and end in .jsonl.zst.age.",
219239
},
220240
},
241+
"store verify": {
242+
Name: "store verify",
243+
Summary: "Verify a generic encrypted tenant-store manifest and artifact boundary.",
244+
JSON: true,
245+
Args: []paramSchema{
246+
{Name: "path", Type: "path", Default: ".", Help: "Tenant store root containing manifest.json."},
247+
},
248+
Flags: []paramSchema{
249+
{Name: "json", Type: "bool", Default: "true", Help: "Print JSON output."},
250+
},
251+
Examples: []string{"fincrawl store verify .", "fincrawl store verify ../tenant-store --json"},
252+
Notes: []string{
253+
"Manifest snapshots must reference existing .jsonl.zst.age or .tar.zst.age files with relative paths.",
254+
"Plaintext archives, local databases, runtime state, logs, reports, screenshots, and transcripts are rejected.",
255+
},
256+
},
221257
"guard": {
222258
Name: "guard",
223259
Summary: "Check commit guardrails for tenant data, plaintext archives, and secret-like values.",
@@ -241,6 +277,9 @@ func describeCommands(command string) (cliSchema, error) {
241277
if command == "" {
242278
return schema, nil
243279
}
280+
if command == "store" {
281+
command = "store verify"
282+
}
244283
described, ok := schema.Commands[command]
245284
if !ok {
246285
return cliSchema{}, output.UsageError{Err: fmt.Errorf("unknown command %q", command)}
@@ -547,11 +586,88 @@ func searchResultField(result store.SearchResult, name string) (any, error) {
547586
return result.UpdatedAt, nil
548587
case "snippet":
549588
return result.Snippet, nil
589+
case "score":
590+
return result.Score, nil
550591
default:
551592
return nil, fmt.Errorf("unknown search field %q", name)
552593
}
553594
}
554595

596+
func projectConversationDetail(detail store.ConversationDetail, fields string) (any, error) {
597+
fields = strings.TrimSpace(fields)
598+
if fields == "" {
599+
return detail, nil
600+
}
601+
names, err := parseConversationFields(fields)
602+
if err != nil {
603+
return nil, err
604+
}
605+
projected := make(map[string]any, len(names))
606+
for _, name := range names {
607+
value, _ := conversationDetailField(detail, name)
608+
projected[name] = value
609+
}
610+
return projected, nil
611+
}
612+
613+
func validateConversationFields(fields string) error {
614+
fields = strings.TrimSpace(fields)
615+
if fields == "" {
616+
return nil
617+
}
618+
_, err := parseConversationFields(fields)
619+
return err
620+
}
621+
622+
func parseConversationFields(fields string) ([]string, error) {
623+
rawNames := strings.Split(fields, ",")
624+
names := make([]string, 0, len(rawNames))
625+
for _, name := range rawNames {
626+
name = strings.TrimSpace(name)
627+
if name == "" {
628+
return nil, errors.New("--fields contains an empty field")
629+
}
630+
if _, err := conversationDetailField(store.ConversationDetail{}, name); err != nil {
631+
return nil, err
632+
}
633+
names = append(names, name)
634+
}
635+
return names, nil
636+
}
637+
638+
func conversationDetailField(detail store.ConversationDetail, name string) (any, error) {
639+
switch name {
640+
case "id":
641+
return detail.ID, nil
642+
case "provider_id":
643+
return detail.ProviderID, nil
644+
case "subject":
645+
return detail.Subject, nil
646+
case "state":
647+
return detail.State, nil
648+
case "assignee":
649+
return detail.Assignee, nil
650+
case "rating":
651+
return detail.Rating, nil
652+
case "fin_status":
653+
return detail.FinStatus, nil
654+
case "participants":
655+
return detail.Participants, nil
656+
case "tags":
657+
return detail.Tags, nil
658+
case "created_at":
659+
return detail.CreatedAt, nil
660+
case "updated_at":
661+
return detail.UpdatedAt, nil
662+
case "snippet":
663+
return detail.Snippet, nil
664+
case "parts":
665+
return detail.Parts, nil
666+
default:
667+
return nil, fmt.Errorf("unknown show field %q", name)
668+
}
669+
}
670+
555671
func hasControlOrSpace(value string) bool {
556672
for _, r := range value {
557673
if unicode.IsControl(r) || unicode.IsSpace(r) {

0 commit comments

Comments
 (0)