Skip to content

Commit 2a0c811

Browse files
altaywtfaltaywtf
committed
docs(oss): prepare public repo docs
1 parent f977d3d commit 2a0c811

9 files changed

Lines changed: 450 additions & 748 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Contributing
2+
3+
`fincrawl` is a generic support-archive CLI. Contributions must keep the core
4+
reusable and must not add tenant-derived data, credentials, local runtime state,
5+
or transcript-derived examples.
6+
7+
## Setup
8+
9+
Use the repo scripts as the source of truth:
10+
11+
```bash
12+
./scripts/smoke
13+
./scripts/verify
14+
```
15+
16+
`mise` is available as a task index for local convenience:
17+
18+
```bash
19+
mise trust mise.toml
20+
mise tasks
21+
```
22+
23+
Use `FINCRAWL_HOME=<scratch-dir>` for manual CLI runs that should not touch the
24+
default user state. Keep scratch homes under ignored paths such as `tmp/` or a
25+
system temp directory.
26+
27+
## Data Boundary
28+
29+
Do not commit:
30+
31+
- Provider tokens, API keys, bearer tokens, `.env` files, or real 1Password item
32+
paths.
33+
- Tenant account IDs, workspace IDs, user/contact IDs, admin names, team names,
34+
tags, or tenant-specific config.
35+
- Real conversation bodies, subjects, summaries, ratings, notes, screenshots,
36+
reports, plaintext snapshots, encrypted tenant snapshots, logs, SQLite stores,
37+
or cache directories.
38+
- Fixtures copied from real conversations, even if redacted.
39+
40+
Committed fixtures must be synthetic and intentionally small.
41+
42+
## Local Live Testing
43+
44+
Live Intercom testing is optional and local. Load credentials through ignored
45+
environment files or direct environment variables. Do not paste command output
46+
containing tenant data into commits, docs, issues, or pull requests.
47+
48+
See [Local live smoke](docs/runbooks/local-live-smoke.md) for the safe local
49+
workflow.
50+
51+
## Verification
52+
53+
Before opening a pull request or committing a non-trivial change, run:
54+
55+
```bash
56+
./scripts/verify
57+
```
58+
59+
For release configuration changes, also run:
60+
61+
```bash
62+
./scripts/release-check
63+
```
64+
65+
For docs-only changes, at minimum check links and run the guard:
66+
67+
```bash
68+
for f in CONTRIBUTING.md SECURITY.md README.md docs/architecture.md docs/roadmap.md docs/runbooks/open-source-readiness.md docs/runbooks/local-live-smoke.md docs/tenant-data-boundary.md docs/references/intercom-api.md skills/fincrawl/SKILL.md; do test -f "$f" || { echo "missing $f"; exit 1; }; done
69+
go run ./cmd/fincrawl guard --json
70+
```
71+
72+
## Commit Style
73+
74+
Use Conventional Commit subjects:
75+
76+
```text
77+
<type>(<scope>): <subject>
78+
```
79+
80+
Examples:
81+
82+
```text
83+
feat(sync): persist updated-since cursors
84+
docs(security): clarify tenant artifact boundary
85+
```
86+
87+
The release pipeline uses Conventional Commit metadata to decide patch releases
88+
on the `0.0.x` bootstrap line.
89+
90+
## Pull Requests
91+
92+
Keep pull requests scoped. Include:
93+
94+
- What changed.
95+
- How it was verified.
96+
- Whether tenant-data boundaries, local state, or generated artifacts are
97+
affected.
98+
- Any remaining manual proof needed.

‎README.md‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,15 @@ encrypted snapshots, and transcript-derived examples do not belong in this repo.
1010

1111
## Start Here
1212

13-
- [Intercom archive MVP](docs/specs/intercom-archive-mvp.md) describes the
14-
product shape, data scope, security model, and done criteria.
15-
- [Bootstrap MVP plan](docs/plans/bootstrap-mvp.md) describes the first Go
16-
implementation slice.
13+
- [Architecture](docs/architecture.md) describes the current CLI, package,
14+
store, sync, artifact, and agent-use shape.
15+
- [Roadmap](docs/roadmap.md) describes what works now and what comes next.
16+
- [Contributing](CONTRIBUTING.md) explains setup, verification, commit style,
17+
and the no-tenant-data contribution boundary.
18+
- [Security](SECURITY.md) explains how to report vulnerabilities without
19+
disclosing credentials or tenant data.
20+
- [Open source readiness](docs/runbooks/open-source-readiness.md) is the
21+
checklist to complete before changing repository visibility.
1722
- [Intercom API reference](docs/references/intercom-api.md) collects public API
1823
links and extraction patterns for agents implementing sync.
1924
- [Tenant data boundary](docs/tenant-data-boundary.md) is the reusable policy
@@ -51,8 +56,8 @@ mise run guard
5156
```
5257

5358
Use `FINCRAWL_HOME=tmp/fincrawl-home` to keep local CLI smoke state inside the
54-
ignored repo `tmp/` directory. The archive command above uses a synthetic public
55-
age recipient for smoke tests only. The CLI may read ignored `.env.local` for
59+
ignored repo `tmp/` directory. Archive examples in this README use a synthetic
60+
public age recipient for smoke tests only. The CLI may read ignored `.env.local` for
5661
`FINCRAWL_AGE_RECIPIENT`; it does not shell out to 1Password.
5762

5863
## Local Live Smoke
@@ -152,3 +157,8 @@ also run semantic-release; Conventional Commit messages decide whether a release
152157
is needed. During bootstrap, releasable commit types advance the patch-only
153158
`0.0.x` line. When a release is published, GoReleaser builds `fincrawl`
154159
binaries for Linux, macOS, and Windows and appends them to the GitHub Release.
160+
161+
## License
162+
163+
No public license has been selected yet. Add an explicit `LICENSE` file before
164+
making this repository public or accepting outside reuse.

‎SECURITY.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Security
2+
3+
`fincrawl` handles support archive mechanics, so security reports may involve
4+
credentials, tenant identifiers, conversation metadata, or transcript content.
5+
Do not disclose those details in public issues, pull requests, commits, logs, or
6+
screenshots.
7+
8+
## Reporting
9+
10+
Use GitHub private vulnerability reporting. It must be enabled before this
11+
repository is made public. If private reporting is unexpectedly unavailable,
12+
open a minimal public issue asking for a private disclosure channel and include
13+
no exploit details, secrets, tenant identifiers, or transcript text.
14+
15+
## Sensitive Data Rules
16+
17+
Never include the following in a report artifact unless a maintainer has
18+
explicitly provided a private channel and asked for the minimum needed excerpt:
19+
20+
- Intercom tokens, API keys, bearer tokens, cloud credentials, or private age or
21+
SSH identities.
22+
- Real 1Password item paths, vault names, or injected environment files.
23+
- Tenant account IDs, workspace IDs, user/contact IDs, admin names, team names,
24+
tags, provider cursors, or conversation URLs.
25+
- Conversation subjects, transcript bodies, summaries, notes, ratings,
26+
screenshots, generated snapshots, SQLite stores, logs, or reports.
27+
28+
Prefer synthetic reproductions. If a live tenant reproduction is unavoidable,
29+
keep it local and describe the vulnerability class without copying tenant data
30+
into the repo.
31+
32+
## Supported Surface
33+
34+
Security review should focus on:
35+
36+
- CLI credential loading and redaction.
37+
- Provider API request construction and rate-limit handling.
38+
- SQLite archive storage, import, search, and raw JSON retention.
39+
- zstd + age artifact generation and import.
40+
- Commit guardrails for generated artifacts, secret-looking values, provider
41+
URLs, and transcript-like content.
42+
- Agent-facing skill guidance that may cause unsafe reads or writes.

‎docs/architecture.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Architecture
2+
3+
Status: active
4+
5+
`fincrawl` is a local-first support conversation archive CLI. The reusable repo
6+
owns generic crawling, storage, search, artifact, guardrail, release, and
7+
agent-facing mechanics. Tenant credentials, tenant config, runtime state,
8+
generated artifacts, logs, reports, screenshots, and transcript-derived examples
9+
stay outside this repo.
10+
11+
## Current Surface
12+
13+
Core commands:
14+
15+
```bash
16+
fincrawl doctor --offline
17+
fincrawl metadata --json
18+
fincrawl status --json
19+
fincrawl sync --fixture testdata/synthetic
20+
fincrawl sync --entities
21+
fincrawl sync --updated-since 2h --limit 50
22+
fincrawl sync --resume
23+
fincrawl sync --conversation <id>
24+
fincrawl search "billing refund" --json
25+
fincrawl archive --fixture testdata/synthetic --recipient <age-recipient> --out tmp/archive.jsonl.zst.age
26+
fincrawl publish --recipient <age-recipient> --out snapshots/local.jsonl.zst.age
27+
fincrawl import --identity <age-identity> --in snapshots/local.jsonl.zst.age
28+
fincrawl guard --json
29+
```
30+
31+
The CLI defaults to JSON output for agent use. Human text output is explicit.
32+
Live provider credentials are optional and local; deterministic verification
33+
uses synthetic fixtures.
34+
35+
## Package Shape
36+
37+
```text
38+
cmd/fincrawl/ CLI entrypoint
39+
internal/cli/ command parsing, JSON output, errors, Agent DX metadata
40+
internal/config/ env and local config loading with redaction
41+
internal/control/ machine-readable command descriptions and safe examples
42+
internal/intercom/ provider API client boundary
43+
internal/syncer/ fixture, entity, exact, and tail sync orchestration
44+
internal/store/ SQLite schema, migrations, export, search, sync state
45+
internal/archive/ canonical JSONL, zstd compression, age encryption
46+
internal/guard/ preflight repository leak checks
47+
internal/lock/ local write lock
48+
```
49+
50+
`openclaw/crawlkit` is used where it fits local crawler mechanics. Provider API
51+
shape, privacy rules, archive format, and support-search semantics stay in
52+
`fincrawl`.
53+
54+
## Store
55+
56+
SQLite is the local archive source of truth. The schema stays migration-friendly
57+
and small:
58+
59+
- `workspaces`
60+
- `conversations`
61+
- `conversation_parts`
62+
- `contacts`
63+
- `admins`
64+
- `teams`
65+
- `tags`
66+
- `conversation_tags`
67+
- `sync_state`
68+
- `raw_blobs`
69+
- `conversation_fts`
70+
71+
Search uses FTS with sanitized queries and a fallback path where needed. Raw
72+
provider JSON is retained locally for replay and migration debugging, but it is
73+
tenant data and must not be committed to this repo.
74+
75+
## Sync
76+
77+
Sync paths are deliberately separate:
78+
79+
- Fixture sync imports synthetic test data and exercises the real store/search
80+
path without credentials.
81+
- Entity sync hydrates read-only provider metadata such as admins, teams, tags,
82+
and capped contacts when scopes allow.
83+
- Exact sync hydrates one conversation by ID.
84+
- Tail sync searches recently updated conversations, stores resumable sync
85+
state, then hydrates exact conversations.
86+
87+
Provider access must use supported APIs or official export paths only. Browser
88+
scraping, undocumented endpoint crawling, credential-sharing workarounds, and
89+
rate-limit bypasses are out of scope.
90+
91+
## Artifacts
92+
93+
Archive output is canonical JSONL streamed through zstd compression and explicit
94+
age encryption. Supported artifact shape:
95+
96+
```text
97+
*.jsonl.zst.age
98+
```
99+
100+
Plaintext archives, databases, logs, snapshots, screenshots, reports, and
101+
transcripts are ignored and blocked by `fincrawl guard`. Encrypted tenant
102+
snapshots are still tenant data and belong outside this repo.
103+
104+
## Agent Use
105+
106+
Agents should load [the repo skill](../skills/fincrawl/SKILL.md) before using a
107+
local archive. It describes binary discovery, read-only search, live sync
108+
boundaries, portable store expectations, and current Agent DX conventions.

0 commit comments

Comments
 (0)