docs: restructure public docs

Author: altaywtf

.github/pull_request_template.md

@@ -0,0 +1,17 @@
+## Summary
+
+
+## Verification
+
+- [ ] `./scripts/verify`
+
+## Data Boundary
+
+- [ ] This change does not commit tenant credentials, tenant identifiers, real
+      transcript data, generated snapshots, logs, reports, screenshots, or
+      transcript-derived examples.
+
+## Release Notes
+
+- [ ] User-facing change worth mentioning
+- [ ] Internal-only change

CONTRIBUTING.md

@@ -6,14 +6,13 @@ or transcript-derived examples.
 
 ## Setup
 
-Use the repo scripts as the source of truth:
+Install Go using the version in `go.mod`, then run the smoke check:
 
 ```bash
 ./scripts/smoke
-./scripts/verify
 ```
 
-`mise` is available as a task index for local convenience:
+`mise` is available as a convenience task index:
 
 ```bash
 mise trust mise.toml
@@ -24,22 +23,15 @@ Use `FINCRAWL_HOME=<scratch-dir>` for manual CLI runs that should not touch the
 default user state. Keep scratch homes under ignored paths such as `tmp/` or a
 system temp directory.
 
-## Data Boundary
-
-Do not commit:
-
-- Provider tokens, API keys, bearer tokens, `.env` files, or real 1Password item
-  paths.
-- Tenant account IDs, workspace IDs, user/contact IDs, admin names, team names,
-  tags, or tenant-specific config.
-- Real conversation bodies, subjects, summaries, ratings, notes, screenshots,
-  reports, plaintext snapshots, encrypted tenant snapshots, logs, SQLite stores,
-  or cache directories.
-- Fixtures copied from real conversations, even if redacted.
+## Run Locally
 
-Committed fixtures must be synthetic and intentionally small.
+Use the source checkout while developing:
 
-## Local Live Testing
+```bash
+go run ./cmd/fincrawl doctor --offline
+go run ./cmd/fincrawl sync --fixture testdata/synthetic
+go run ./cmd/fincrawl search Morgan --fields provider_id,subject,updated_at
+```
 
 Live Intercom testing is optional and local. Load credentials through ignored
 environment files or direct environment variables. Do not paste command output
@@ -48,9 +40,9 @@ containing tenant data into commits, docs, issues, or pull requests.
 See [Local live smoke](docs/runbooks/local-live-smoke.md) for the safe local
 workflow.
 
-## Verification
+## Validation
 
-Before opening a pull request or committing a non-trivial change, run:
+Run the full local gate before opening a pull request:
 
 ```bash
 ./scripts/verify
@@ -62,13 +54,45 @@ For release configuration changes, also run:
 ./scripts/release-check
 ```
 
-For docs-only changes, at minimum check links and run the guard:
+For docs-only changes, at minimum check the linked doc surface and run the repo
+guard:
 
 ```bash
-for f in CONTRIBUTING.md SECURITY.md README.md docs/architecture.md docs/roadmap.md docs/runbooks/open-source-readiness.md docs/runbooks/local-live-smoke.md docs/tenant-data-boundary.md docs/references/intercom-api.md skills/fincrawl/SKILL.md; do test -f "$f" || { echo "missing $f"; exit 1; }; done
+for f in README.md CONTRIBUTING.md SECURITY.md docs/architecture.md docs/roadmap.md docs/distribution.md docs/runbooks/open-source-readiness.md docs/runbooks/local-live-smoke.md docs/tenant-data-boundary.md docs/references/intercom-api.md skills/fincrawl/SKILL.md; do test -f "$f" || { echo "missing $f"; exit 1; }; done
 go run ./cmd/fincrawl guard --json
 ```
 
+## Data Boundary
+
+Committed fixtures must be synthetic and intentionally small.
+
+Do not commit:
+
+- Provider tokens, API keys, bearer tokens, `.env` files, or real 1Password item
+  paths.
+- Tenant account IDs, workspace IDs, user/contact IDs, admin names, team names,
+  tags, or tenant-specific config.
+- Real conversation bodies, subjects, summaries, ratings, notes, screenshots,
+  reports, plaintext snapshots, encrypted tenant snapshots, logs, SQLite stores,
+  or cache directories.
+- Fixtures copied from real conversations, even if redacted.
+
+Use [Tenant data boundary](docs/tenant-data-boundary.md) as the source of truth
+when a change touches credentials, live sync, snapshots, logs, fixtures, or
+agent-facing examples.
+
+## Pull Requests
+
+Keep pull requests scoped. Include:
+
+- What changed.
+- How it was verified.
+- Whether tenant-data boundaries, local state, or generated artifacts are
+  affected.
+- Any remaining manual proof needed.
+
+The pull request template asks for the same evidence in a compact form.
+
 ## Commit Style
 
 Use Conventional Commit subjects:
@@ -85,14 +109,5 @@ docs(security): clarify tenant artifact boundary
 ```
 
 The release pipeline uses Conventional Commit metadata to decide patch releases
-on the `0.0.x` bootstrap line.
-
-## Pull Requests
-
-Keep pull requests scoped. Include:
-
-- What changed.
-- How it was verified.
-- Whether tenant-data boundaries, local state, or generated artifacts are
-  affected.
-- Any remaining manual proof needed.
+on the `0.0.x` bootstrap line. See [Distribution](docs/distribution.md) for the
+release model.

README.md

@@ -1,163 +1,138 @@
 # fincrawl
 
-`fincrawl` is a reusable local-first crawler/archive tool for support
-conversation systems, starting with Intercom and Fin.
-
-The project is the generic crawler core: CLI, schemas, local archive mechanics,
-search, encryption pipeline, fake fixtures, tests, and docs. Tenant credentials,
-runtime config, crawl state, logs, reports, screenshots, plaintext snapshots,
-encrypted snapshots, and transcript-derived examples do not belong in this repo.
-
-## Start Here
-
-- [Architecture](docs/architecture.md) describes the current CLI, package,
-  store, sync, artifact, and agent-use shape.
-- [Roadmap](docs/roadmap.md) describes what works now and what comes next.
-- [Contributing](CONTRIBUTING.md) explains setup, verification, commit style,
-  and the no-tenant-data contribution boundary.
-- [Security](SECURITY.md) explains how to report vulnerabilities without
-  disclosing credentials or tenant data.
-- [Open source readiness](docs/runbooks/open-source-readiness.md) is the
-  checklist to complete before changing repository visibility.
-- [Intercom API reference](docs/references/intercom-api.md) collects public API
-  links and extraction patterns for agents implementing sync.
-- [Tenant data boundary](docs/tenant-data-boundary.md) is the reusable policy
-  for credentials, real artifacts, manual tenant testing, and committed
-  fixtures.
-- [Local live smoke](docs/runbooks/local-live-smoke.md) shows how to exercise
-  read-only Intercom calls without committing tenant state.
-
-## Development Boundary
-
-Committed tests and fixtures must be fake or synthetic. Real tenant crawl state
-and generated artifacts belong in tenant-controlled private storage outside this
-repository, even when encrypted.
-
-## Current Offline Slice
-
-The bootstrapped CLI runs without live Intercom credentials and stores only
-synthetic fixture data during deterministic checks:
+Local-first support conversation archive CLI for Intercom workspaces, including
+conversation metadata that Intercom exposes for Fin participation.
+
+`fincrawl` syncs support conversations into a private local SQLite archive,
+searches them without going back through the Intercom UI, and exports portable
+snapshots as compressed age-encrypted JSONL.
+
+This repository is the generic crawler core. Tenant credentials, tenant config,
+runtime state, crawl output, logs, reports, screenshots, plaintext snapshots,
+encrypted tenant snapshots, and transcript-derived examples do not belong here.
+
+## Install
+
+Download a binary from the latest
+[GitHub Release](https://github.com/uinaf/fincrawl/releases), or install from
+source with Go:
 
 ```bash
-./scripts/smoke
-./scripts/verify
+go install github.com/uinaf/fincrawl/cmd/fincrawl@latest
 ```
 
-`mise` is available as a convenience task index for ad-hoc local work. The
-underlying scripts remain the source of truth for verification:
+To work from a checkout:
 
 ```bash
-mise trust mise.toml
-mise tasks
-mise run smoke
-mise run fixture-loop
-mise run verify
-mise run guard
+go run ./cmd/fincrawl version
+./scripts/smoke
 ```
 
-Use `FINCRAWL_HOME=tmp/fincrawl-home` to keep local CLI smoke state inside the
-ignored repo `tmp/` directory. Archive examples in this README use a synthetic
-public age recipient for smoke tests only. The CLI may read ignored `.env.local` for
-`FINCRAWL_AGE_RECIPIENT`; it does not shell out to 1Password.
-
-## Local Live Smoke
+## Quick Start
 
-Live sync is intentionally manual and local. Use a read-only Intercom app token
-from ignored `.env.local`; do not write tenant config or generated artifacts
-inside this repository. The safe default smoke hydrates admins, teams, and tags
-only; contact listing and conversation hydration are explicit opt-ins.
+Keep local state outside tracked paths:
 
 ```bash
-cp .env.local.example .env.local.tpl
-op inject -i .env.local.tpl -o .env.local
-chmod 600 .env.local
-./scripts/local-live-smoke
+export FINCRAWL_HOME=/tmp/fincrawl-home
+fincrawl doctor --offline
 ```
 
-For contact scope proof, keep the list capped:
+Try the archive path with synthetic fixtures from a source checkout:
 
 ```bash
-FINCRAWL_LIVE_CONTACT_LIMIT=10 ./scripts/local-live-smoke
+fincrawl sync --fixture testdata/synthetic
+fincrawl search "billing refund" --fields provider_id,subject,updated_at
 ```
 
-For bounded conversation proof, prefer a short window and explicit limit:
+For live Intercom access, use a tenant-authorized read-only token from your
+environment or ignored local env file:
 
 ```bash
-FINCRAWL_LIVE_UPDATED_SINCE=2h FINCRAWL_LIVE_CONVERSATION_LIMIT=5 ./scripts/local-live-smoke
+fincrawl sync --entities
+fincrawl sync --updated-since 2h --limit 50
+fincrawl search "login code expired" --fields provider_id,subject,updated_at
 ```
 
-See [Local live smoke](docs/runbooks/local-live-smoke.md) for the full local
-workflow and cleanup notes.
+Use exact hydration when you already know a conversation ID:
+
+```bash
+fincrawl sync --conversation <intercom-conversation-id>
+```
 
-## Current Local Slice
+## Snapshots
 
-The current local slice focuses on entity hydration, useful local search, and
-local encrypted snapshot portability:
+Snapshots are compressed and encrypted before storage:
 
-- Hydrate and normalize Intercom admins, teams, tags, and capped contacts/users
-  where the tenant-authorized token exposes them.
-- Enrich SQLite and FTS with participant, assignee, tag, rating, state, and
-  Fin-like metadata while keeping provider-specific raw JSON for replay.
-- Provide local smoke checks for read-only scopes without writing tenant config,
-  logs, snapshots, or generated artifacts into this repo.
-- Publish local SQLite state as compressed age-encrypted JSONL with
-  `fincrawl publish`.
-- Import compressed age-encrypted JSONL into a fresh local SQLite archive with
-  `fincrawl import`.
+```bash
+fincrawl publish \
+  --recipient <age-recipient-or-ssh-public-key> \
+  --out snapshots/local.jsonl.zst.age \
+  --dry-run
+```
 
-The local publish/import loop is the current portability path. Remote
-publish/push/subscribe flows are intentionally deferred while tenant-controlled
-local usage proves the archive is useful.
+Import validates and hydrates a local SQLite archive:
 
-The default Intercom API version is `2.15`. Set
-`FINCRAWL_INTERCOM_BASE_URL` only for a regional Intercom API host and
-`FINCRAWL_INTERCOM_VERSION` only when intentionally testing another supported
-API version.
+```bash
+FINCRAWL_AGE_IDENTITY=<age-identity> \
+  fincrawl import --in snapshots/local.jsonl.zst.age --dry-run
+```
 
-## Agent Workflow
+Plaintext archive output is local scratch data only. Tenant encrypted snapshots
+still belong in tenant-controlled private storage, not in this repository.
 
-Agents that need to use `fincrawl` as a local support archive should load the
-repo skill at [skills/fincrawl/SKILL.md](skills/fincrawl/SKILL.md). It covers
-local search, bounded live sync, tenant-data boundaries, and current CLI Agent
-DX guidance.
+## Commands
 
-Useful agent-facing commands:
+The CLI defaults to JSON output for agents and automation:
 
 ```bash
-fincrawl describe search
-fincrawl search "login code expired" --fields provider_id,subject,updated_at
-fincrawl search "login code expired" --fields provider_id,subject,updated_at --ndjson
-fincrawl sync --updated-since 2h --limit 50 --dry-run
-fincrawl archive --fixture testdata/synthetic --recipient age1n9zrm0rcxehv7cm55uqw27v9cguz4ev5dtyl7kxkn3vdpvap94ds2gn6rl --out tmp/snapshot.jsonl.zst.age --dry-run
-fincrawl publish --recipient <age-recipient> --out snapshots/local.jsonl.zst.age --dry-run
-FINCRAWL_AGE_IDENTITY=<age-identity> fincrawl import --in snapshots/local.jsonl.zst.age --dry-run
+fincrawl describe --json
+fincrawl describe search --json
+fincrawl status --json
+fincrawl guard --json
 ```
 
-CLI output defaults to JSON, including structured error envelopes. Use
-`--json=false` only for human-oriented text output.
-
-`./scripts/smoke` runs a fast offline CLI proof against synthetic fixtures.
-`./scripts/verify` runs the full local gate: module tidy check, tests, vet, race
-tests, smoke, guardrails, and whitespace checks.
-`./scripts/release-check` validates the GoReleaser release config.
-
-Enable the repo hook path if you want the local pre-push gate:
+Common flows:
+
+| Need | Command |
+| --- | --- |
+| Check local config | `fincrawl doctor --offline` |
+| Sync metadata | `fincrawl sync --entities` |
+| Sync a recent window | `fincrawl sync --updated-since 2h --limit 50` |
+| Hydrate one conversation | `fincrawl sync --conversation <id>` |
+| Search local archive | `fincrawl search "<query>" --fields provider_id,subject,updated_at` |
+| Export encrypted snapshot | `fincrawl publish --recipient <recipient> --out snapshots/local.jsonl.zst.age` |
+| Import encrypted snapshot | `fincrawl import --identity <identity> --in snapshots/local.jsonl.zst.age` |
+| Check repo guardrails | `fincrawl guard --json` |
+
+## Docs
+
+| Need | Read |
+| --- | --- |
+| Understand the CLI and storage design | [Architecture](docs/architecture.md) |
+| See what exists and what is next | [Roadmap](docs/roadmap.md) |
+| Keep tenant data out of the repo | [Tenant data boundary](docs/tenant-data-boundary.md) |
+| Run bounded live Intercom checks | [Local live smoke](docs/runbooks/local-live-smoke.md) |
+| Work on releases and CI | [Distribution](docs/distribution.md) |
+| Implement Intercom sync safely | [Intercom API reference](docs/references/intercom-api.md) |
+| Use fincrawl as an agent skill | [Agent guide](skills/fincrawl/SKILL.md) |
+| Contribute changes | [Contributing](CONTRIBUTING.md) |
+| Report a vulnerability | [Security](SECURITY.md) |
+
+## Contributing
+
+Run the local gate before sending changes:
 
 ```bash
-git config core.hooksPath .git-hooks
+./scripts/verify
 ```
 
-The pre-push hook runs tests, vet, smoke, and guardrails. The commit-msg hook
-enforces Conventional Commit subjects for semantic-release.
+See [Contributing](CONTRIBUTING.md) for setup, validation, commit style, and the
+tenant-data contribution boundary.
 
-## Release
+## Security
 
-Pushes and pull requests run the verify job in GitHub Actions. Pushes to `main`
-also run semantic-release; Conventional Commit messages decide whether a release
-is needed. During bootstrap, releasable commit types advance the patch-only
-`0.0.x` line. When a release is published, GoReleaser builds `fincrawl`
-binaries for Linux, macOS, and Windows and appends them to the GitHub Release.
+Report vulnerabilities privately. See [Security](SECURITY.md).
 
 ## License
 
-`fincrawl` is released under the [MIT License](LICENSE).
+MIT. See [License](LICENSE).