Commit 5a5a32d
Forward PRD completion (#1)
* feat(threads-ingest): Normalizer + orchestration pipeline
- Normalizer trait with OfficialNormalizer impl mapping graph.threads.net
JSON into threads-core's stable Post/User/Media/Edge model.
- Handles /me, single post, and {data, paging.cursors.after} pagination
envelope. Respects root_hint for reply threads, synthesizes author from
owner.id or username fallback, walks CAROUSEL_ALBUM children for media.
- Retains full raw JSON on Post.raw per PRD for replay/re-normalization.
- Ingestor orchestrator: uuid-v4 fetch_run_id, pagination loop, per-run
PostId HashSet dedup, 100-at-a-time batched upserts, FetchRun start/end
recording (success + error paths), tracing::info milestones.
- StoreWrite trait keeps ingest decoupled from the concrete store for
tests; `impl StoreWrite for threads_store::Store` bridges them (Phase 2).
13 tests: 9 normalizer fixture tests + 4 orchestrator MockProvider/MockStore
integration tests (deduplication, run-start/end recording, single-page).
* feat(threads-provider-official): OAuth + HTTP client + Provider impl
- Config (env + file) with app_id/app_secret/redirect_uri/access_token.
- OAuth 2.0 authorization-code flow:
- authorize_url() targets threads.net/oauth/authorize with comma-joined
scopes and CSRF state.
- exchange_code() POSTs form to graph.threads.net/oauth/access_token.
- upgrade_to_long_lived() via th_exchange_token.
- refresh_long_lived() via th_refresh_token.
- CallbackServer: binds 127.0.0.1:0, parses first GET, validates state,
returns a "you may close this window" HTML page. Pure tokio + hand-
rolled percent-decoding; no extra HTTP server crate.
- TokenStore: keyring primary (service threads-cli), JSON file fallback at
~/.config/threads-cli/token.json. Expiry-aware.
- HttpClient (reqwest): auto access_token param, field= projection, 429
+ x-app-usage>=90% exponential backoff with jitter (cap 30s, max 5),
maps status to threads_core::Error variants.
- DTOs with #[serde(default)] for Meta's often-missing fields.
- OfficialProvider implements threads_core::Provider, driving endpoints
off the versioned manifest (substituting {post-id}).
17 unit tests covering OAuth URL building, request parsing, percent
decoding, callback bind, DTO parsing, near-limit detection, backoff,
DTO->Post mapping with root_hint.
* feat(threads-cli): clap subcommands wired end-to-end
- clap derive CLI with --config, --db, --verbose, --format globals.
- Subcommands: init, auth {login, status, logout}, ingest {me, thread},
show [--thread], search, export [--out PATH].
- init: prompts for App ID/Secret/Redirect URI and writes
~/.config/threads-cli/config.toml (or --force to overwrite).
- auth login (Phase 2): binds local CallbackServer (127.0.0.1:0), opens
browser via `open`, runs full OAuth flow (authorize -> exchange ->
long-lived upgrade), persists Token via TokenStore.
- auth status/logout: inspects or clears the keyring + file token.
- ingest me/thread (Phase 2): builds OfficialProvider from manifest +
token, spins up an Ingestor with OfficialNormalizer, reports FetchRun
summary.
- show/search/export (Phase 3): reads directly from threads_store; show
uses thread_rooted_at (recursive CTE) when --thread is set; search
uses FTS5 MATCH; export streams render_posts to stdout or --out.
- Output module: Human, Json, Jsonl, Csv formatters (round-tripping).
- Config: env > file > defaults precedence, with defaults resolved via
dirs::{config_dir, data_local_dir}.
- Embedded manifest via include_str!("../../../manifests/official_v1.toml")
— no runtime file lookup needed.
- Last cleanups: clippy warnings addressed (from_str, match->if let,
write_literal), unused-field allows where intentional.
9 CLI-level tests (clap structure debug_assert, env-override precedence,
config save/load roundtrip, init --force gate, render JSON/JSONL/CSV/Human).
Build + test gate: cargo test --workspace passes 59 tests, cargo clippy
--workspace --all-targets is clean.
* fix(auth): support manual-paste OAuth for Meta's HTTPS-only redirects
Meta's Threads API rejects http:// redirect URIs (error 1349187
"Insecure Login Blocked") even on localhost, so the auto-listener
flow on 127.0.0.1 can't complete.
- auth login now picks a flow based on the configured redirect_uri:
- http://127.0.0.1 / http://localhost -> local CallbackServer (kept
for other OAuth2 providers and future-proofing)
- anything else -> manual paste: print authorize URL, open browser,
user pastes the redirected URL (or bare code) back into stdin
- parse_code_from_input accepts either a full URL with ?code=&state=
or a bare code string; state mismatch aborts the flow
- init now explains the Meta redirect-URI requirement and warns on
non-loopback http:// URIs
4 new parse tests cover full-URL, bare-code, empty-input, and
URL-without-code cases.
* fix(auth): bind to configured port instead of random OS-assigned one
The local-listener flow was binding 127.0.0.1:0 (OS-assigned) and
overwriting the configured redirect_uri with the random port. Meta
requires redirect_uri byte-equality with the app-dashboard entry, so
any loopback-with-random-port attempt was getting rejected.
- Add CallbackServer::bind_to_uri(&str) that parses the configured URI
and binds to its exact host+port. Rejects https:// (requires TLS),
non-loopback hosts, or missing ports.
- login_local_listener prefers bind_to_uri when the URI has a port;
falls back to OS-assigned ONLY when no port is configured, with a
warning — this is useful for providers that accept port-agnostic
loopback but is now explicit, not accidental.
- init now requires a non-empty redirect URI (no default — Meta
demands exact match, so guessing is worse than asking) and warns
when a loopback http:// URI omits a port.
5 new auth tests: bind_to_uri preserves exact port, rejects https://,
rejects non-loopback, requires a port; existing random-port flow still
covered.
* fix(auth): accept Meta's integer user_id in OAuth token response
Real-world Meta /oauth/access_token returns user_id as a JSON integer
that overflows i32 (e.g. 26490227934002266), while most Graph API
responses return IDs as strings. Our TokenResponse declared
user_id: Option<String>, so parsing failed with:
"invalid type: integer 26490227934002266, expected a string"
Fix: custom deserializer accepting String, i64, or u64 and normalizing
to String so downstream code has one type.
3 new tests: integer user_id, string user_id, absent user_id.
* feat: XDG paths cross-platform + list_posts for enumeration
Two fixes on a single theme ("v1 CLI polish"):
1. XDG-compliant paths everywhere.
The dirs crate returns platform-native dirs, which on macOS means
~/Library/Application Support. For a CLI that users run on both Linux
and macOS, that's surprising. Switch to XDG semantics:
- config/token: $XDG_CONFIG_HOME or ~/.config
- store.db: $XDG_DATA_HOME or ~/.local/share
Applied in both threads-cli::CliConfig and
threads-provider-official::TokenStore so paths stay aligned.
2. Store::list_posts(limit) and wire it through.
FTS5 MATCH requires a valid query token; "*" alone returns 0 rows,
and our export command was mistakenly using search_text("*") to
enumerate. Added a plain SELECT ... ORDER BY fetched_at DESC path.
- export uses list_posts (includes posts with NULL/empty text)
- search treats "*" / empty as a convenience alias to list_posts
- search with a real query still goes through FTS5 MATCH + BM25
* fix(ingest): parse Meta's non-RFC3339 timestamps; also walk /me/replies
Two bugs surfaced on the first real-world ingest:
1. created_at was NULL for all 594 ingested posts.
Meta returns timestamps like "2026-04-24T18:15:44+0000" — valid ISO
8601 but NOT RFC 3339 (which mandates a colon in the TZ offset).
parse_timestamp() only tried RFC 3339 and silently dropped everything.
Now tries RFC 3339 first, then falls back to "%Y-%m-%dT%H:%M:%S%z"
which handles the colonless form. 3 new tests guard this.
2. ingest_me only called /me/threads — replies to other posts were
never fetched. Added an optional fetch_my_replies() method on the
Provider trait (default impl returns empty so the trait stays
backward-compatible for future providers), implemented it in
OfficialProvider against the manifest's me/replies edge, and extended
the orchestrator's ingest_me to walk a second paginated loop for
replies after the threads loop — deduplicated via the same HashSet,
batched into the same upsert flush.
* fix(threads-provider-official): chmod 0600 on fallback token file
Addresses Codex adversarial-review finding #2.
When the keyring is unavailable the token is persisted to
~/.config/threads-cli/token.json. The previous code used fs::write, which
obeyed the process umask (commonly 0644) and exposed a long-lived Threads
bearer token to other local users and backup/indexing processes.
Unix hardening:
- Parent dir created with DirBuilder::mode(0o700).
- File opened via OpenOptions::mode(0o600) and re-chmod'd post-write to
tighten any pre-existing loose mode.
- Pre-existing parent dir with group/world bits is tightened to 0700 on
save.
- On load, a tracing::warn! is emitted if the file is group/world-readable
(no refusal — would break existing users; just surface the issue).
Windows: relies on NTFS ACL inheritance from the user profile; keyring is
the primary store there anyway.
3 new unix-only tests: file-is-0600, dir-is-0700, pre-existing-loose-perms
are tightened for both dir and file.
* fix(threads-store): drop stale edges on post re-upsert
Addresses Codex adversarial-review finding #1.
upsert_post_tx already deleted media/urls/mentions for the post before
re-inserting, but it left existing `edges` rows alone. Reingesting a post
with a changed parent_id/root_id/mention/quote — or with any of those
cleared — would leave stale rows behind, corrupting recursive thread
traversal.
Fix: inside the same transaction, DELETE FROM edges WHERE from_id = ?1
AND kind IN ('reply','root','mention','quote') before the INSERT OR
IGNORE inserts. The kind IN (...) filter lets future non-managed edge
kinds coexist if they're ever added.
Tests: reupsert_without_parent_drops_stale_edges (reply+root+mention ->
0 edges on top-level re-upsert), reupsert_replaces_mention_edges (M1 ->
M2 swap leaves only the M2 edge). Added test-only probes in query.rs
(test_only_count_edges_from / test_only_edge_target) reaching through
Store::raw_conn() so the assertions don't need a public raw-SQL surface.
* fix(threads-ingest): persist root post in ingest_thread
Addresses Codex adversarial-review finding #3.
run_ingest_thread used to paginate fetch_replies only. If the root post
wasn't already in the store, `threads-cli ingest thread <id>` recorded
a successful run while omitting the requested root entirely; for a
thread with no replies it stored ZERO posts and still reported success.
Fix: use Provider::fetch_thread, which drives the manifest's
`post/conversation` edge and returns root + all descendants in one call.
The orchestrator keeps dedup-via-HashSet, batched upserts, and the
FetchRun start/end recording; only the fetch pattern changes.
3 new regression tests in ingest_tests.rs:
- ingest_thread_persists_root_even_with_no_replies: 1 post stored,
run.posts_fetched == 1, no error.
- ingest_thread_persists_root_and_descendants: root + 2 replies ->
3 posts stored.
- ingest_thread_empty_result_still_records_run_end: fetch_thread empty
(root not found) still closes the run cleanly with 0 posts.
MockProvider gains `with_thread(Vec<Post>)` so tests can seed
fetch_thread output.
* chore: codex adversarial review artifact + clippy nit
- Add docs/reviews/codex-adversarial-2026-04-24.md so the review stays
discoverable alongside the fixes it prompted.
- Collapse nested if in init.rs per clippy::collapsible_if.
After this commit, cargo test --workspace is 81 green; cargo clippy
--workspace --all-targets is clean.
* feat(ingest): `ingest engagement` crawls replies-to-me via BFS
Adds the core "collect the reply tree under my posts/replies" workflow
called out as a top PRD use-case. When somebody replies to one of your
posts — and then someone replies to that reply, and a long reply forks
into parent → child → grandchild siblings — those posts aren't reachable
via `/me/threads` or `/me/replies`. This command fills the gap.
Algorithm:
- Seed = every post in the local store where `author_id = me.id`
(`ingest me` must have run at least once).
- BFS: for each seed, paginate `fetch_replies(seed_id)`; for each reply,
recurse until `--depth` hops below the seed. Shared `HashSet<PostId>`
dedupes across seeds and across levels so the same post is never
fetched twice, even when two of your posts share a descendant.
- Batched upserts (100 at a time) tagged with the run's `fetch_run_id`
for provenance; FetchRun start/end recorded as usual.
CLI surface:
threads-cli ingest engagement [--depth N]
Default depth is 8 (real Threads conversations rarely exceed 4–5).
Uses `/{post-id}/replies` (top-level children + pagination) per Meta's
docs — intentionally NOT `/{post-id}/conversation`, which returns the
whole thread from root and wastes calls/storage on ancestors + siblings.
Infra:
- threads-store: `posts_by_author(UserId) -> Vec<PostId>` (direct SELECT).
- StoreWrite: matching trait method + MockStore impl for tests.
- 4 new orchestrator tests: direct replies, 3-level recursion, depth cap,
cross-seed dedup. Total workspace tests: 81 → 85.
---------
Co-authored-by: Chris George <chrisageorge@gmail.com>1 parent d74499c commit 5a5a32d
38 files changed
Lines changed: 4254 additions & 55 deletions
File tree
- .claude
- crates
- threads-cli
- src
- commands
- threads-core/src
- threads-ingest
- src
- tests
- fixtures
- threads-manifest/src
- threads-provider-official
- src
- threads-store/src
- docs/reviews
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | | - | |
35 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
0 commit comments