Skip to content

Commit 5a5a32d

Browse files
chrisgeoChris George
andauthored
Forward PRD completion (#1)
* feat(threads-ingest): Normalizer + orchestration pipeline - Normalizer trait with OfficialNormalizer impl mapping graph.threads.net JSON into threads-core's stable Post/User/Media/Edge model. - Handles /me, single post, and {data, paging.cursors.after} pagination envelope. Respects root_hint for reply threads, synthesizes author from owner.id or username fallback, walks CAROUSEL_ALBUM children for media. - Retains full raw JSON on Post.raw per PRD for replay/re-normalization. - Ingestor orchestrator: uuid-v4 fetch_run_id, pagination loop, per-run PostId HashSet dedup, 100-at-a-time batched upserts, FetchRun start/end recording (success + error paths), tracing::info milestones. - StoreWrite trait keeps ingest decoupled from the concrete store for tests; `impl StoreWrite for threads_store::Store` bridges them (Phase 2). 13 tests: 9 normalizer fixture tests + 4 orchestrator MockProvider/MockStore integration tests (deduplication, run-start/end recording, single-page). * feat(threads-provider-official): OAuth + HTTP client + Provider impl - Config (env + file) with app_id/app_secret/redirect_uri/access_token. - OAuth 2.0 authorization-code flow: - authorize_url() targets threads.net/oauth/authorize with comma-joined scopes and CSRF state. - exchange_code() POSTs form to graph.threads.net/oauth/access_token. - upgrade_to_long_lived() via th_exchange_token. - refresh_long_lived() via th_refresh_token. - CallbackServer: binds 127.0.0.1:0, parses first GET, validates state, returns a "you may close this window" HTML page. Pure tokio + hand- rolled percent-decoding; no extra HTTP server crate. - TokenStore: keyring primary (service threads-cli), JSON file fallback at ~/.config/threads-cli/token.json. Expiry-aware. - HttpClient (reqwest): auto access_token param, field= projection, 429 + x-app-usage>=90% exponential backoff with jitter (cap 30s, max 5), maps status to threads_core::Error variants. - DTOs with #[serde(default)] for Meta's often-missing fields. - OfficialProvider implements threads_core::Provider, driving endpoints off the versioned manifest (substituting {post-id}). 17 unit tests covering OAuth URL building, request parsing, percent decoding, callback bind, DTO parsing, near-limit detection, backoff, DTO->Post mapping with root_hint. * feat(threads-cli): clap subcommands wired end-to-end - clap derive CLI with --config, --db, --verbose, --format globals. - Subcommands: init, auth {login, status, logout}, ingest {me, thread}, show [--thread], search, export [--out PATH]. - init: prompts for App ID/Secret/Redirect URI and writes ~/.config/threads-cli/config.toml (or --force to overwrite). - auth login (Phase 2): binds local CallbackServer (127.0.0.1:0), opens browser via `open`, runs full OAuth flow (authorize -> exchange -> long-lived upgrade), persists Token via TokenStore. - auth status/logout: inspects or clears the keyring + file token. - ingest me/thread (Phase 2): builds OfficialProvider from manifest + token, spins up an Ingestor with OfficialNormalizer, reports FetchRun summary. - show/search/export (Phase 3): reads directly from threads_store; show uses thread_rooted_at (recursive CTE) when --thread is set; search uses FTS5 MATCH; export streams render_posts to stdout or --out. - Output module: Human, Json, Jsonl, Csv formatters (round-tripping). - Config: env > file > defaults precedence, with defaults resolved via dirs::{config_dir, data_local_dir}. - Embedded manifest via include_str!("../../../manifests/official_v1.toml") — no runtime file lookup needed. - Last cleanups: clippy warnings addressed (from_str, match->if let, write_literal), unused-field allows where intentional. 9 CLI-level tests (clap structure debug_assert, env-override precedence, config save/load roundtrip, init --force gate, render JSON/JSONL/CSV/Human). Build + test gate: cargo test --workspace passes 59 tests, cargo clippy --workspace --all-targets is clean. * fix(auth): support manual-paste OAuth for Meta's HTTPS-only redirects Meta's Threads API rejects http:// redirect URIs (error 1349187 "Insecure Login Blocked") even on localhost, so the auto-listener flow on 127.0.0.1 can't complete. - auth login now picks a flow based on the configured redirect_uri: - http://127.0.0.1 / http://localhost -> local CallbackServer (kept for other OAuth2 providers and future-proofing) - anything else -> manual paste: print authorize URL, open browser, user pastes the redirected URL (or bare code) back into stdin - parse_code_from_input accepts either a full URL with ?code=&state= or a bare code string; state mismatch aborts the flow - init now explains the Meta redirect-URI requirement and warns on non-loopback http:// URIs 4 new parse tests cover full-URL, bare-code, empty-input, and URL-without-code cases. * fix(auth): bind to configured port instead of random OS-assigned one The local-listener flow was binding 127.0.0.1:0 (OS-assigned) and overwriting the configured redirect_uri with the random port. Meta requires redirect_uri byte-equality with the app-dashboard entry, so any loopback-with-random-port attempt was getting rejected. - Add CallbackServer::bind_to_uri(&str) that parses the configured URI and binds to its exact host+port. Rejects https:// (requires TLS), non-loopback hosts, or missing ports. - login_local_listener prefers bind_to_uri when the URI has a port; falls back to OS-assigned ONLY when no port is configured, with a warning — this is useful for providers that accept port-agnostic loopback but is now explicit, not accidental. - init now requires a non-empty redirect URI (no default — Meta demands exact match, so guessing is worse than asking) and warns when a loopback http:// URI omits a port. 5 new auth tests: bind_to_uri preserves exact port, rejects https://, rejects non-loopback, requires a port; existing random-port flow still covered. * fix(auth): accept Meta's integer user_id in OAuth token response Real-world Meta /oauth/access_token returns user_id as a JSON integer that overflows i32 (e.g. 26490227934002266), while most Graph API responses return IDs as strings. Our TokenResponse declared user_id: Option<String>, so parsing failed with: "invalid type: integer 26490227934002266, expected a string" Fix: custom deserializer accepting String, i64, or u64 and normalizing to String so downstream code has one type. 3 new tests: integer user_id, string user_id, absent user_id. * feat: XDG paths cross-platform + list_posts for enumeration Two fixes on a single theme ("v1 CLI polish"): 1. XDG-compliant paths everywhere. The dirs crate returns platform-native dirs, which on macOS means ~/Library/Application Support. For a CLI that users run on both Linux and macOS, that's surprising. Switch to XDG semantics: - config/token: $XDG_CONFIG_HOME or ~/.config - store.db: $XDG_DATA_HOME or ~/.local/share Applied in both threads-cli::CliConfig and threads-provider-official::TokenStore so paths stay aligned. 2. Store::list_posts(limit) and wire it through. FTS5 MATCH requires a valid query token; "*" alone returns 0 rows, and our export command was mistakenly using search_text("*") to enumerate. Added a plain SELECT ... ORDER BY fetched_at DESC path. - export uses list_posts (includes posts with NULL/empty text) - search treats "*" / empty as a convenience alias to list_posts - search with a real query still goes through FTS5 MATCH + BM25 * fix(ingest): parse Meta's non-RFC3339 timestamps; also walk /me/replies Two bugs surfaced on the first real-world ingest: 1. created_at was NULL for all 594 ingested posts. Meta returns timestamps like "2026-04-24T18:15:44+0000" — valid ISO 8601 but NOT RFC 3339 (which mandates a colon in the TZ offset). parse_timestamp() only tried RFC 3339 and silently dropped everything. Now tries RFC 3339 first, then falls back to "%Y-%m-%dT%H:%M:%S%z" which handles the colonless form. 3 new tests guard this. 2. ingest_me only called /me/threads — replies to other posts were never fetched. Added an optional fetch_my_replies() method on the Provider trait (default impl returns empty so the trait stays backward-compatible for future providers), implemented it in OfficialProvider against the manifest's me/replies edge, and extended the orchestrator's ingest_me to walk a second paginated loop for replies after the threads loop — deduplicated via the same HashSet, batched into the same upsert flush. * fix(threads-provider-official): chmod 0600 on fallback token file Addresses Codex adversarial-review finding #2. When the keyring is unavailable the token is persisted to ~/.config/threads-cli/token.json. The previous code used fs::write, which obeyed the process umask (commonly 0644) and exposed a long-lived Threads bearer token to other local users and backup/indexing processes. Unix hardening: - Parent dir created with DirBuilder::mode(0o700). - File opened via OpenOptions::mode(0o600) and re-chmod'd post-write to tighten any pre-existing loose mode. - Pre-existing parent dir with group/world bits is tightened to 0700 on save. - On load, a tracing::warn! is emitted if the file is group/world-readable (no refusal — would break existing users; just surface the issue). Windows: relies on NTFS ACL inheritance from the user profile; keyring is the primary store there anyway. 3 new unix-only tests: file-is-0600, dir-is-0700, pre-existing-loose-perms are tightened for both dir and file. * fix(threads-store): drop stale edges on post re-upsert Addresses Codex adversarial-review finding #1. upsert_post_tx already deleted media/urls/mentions for the post before re-inserting, but it left existing `edges` rows alone. Reingesting a post with a changed parent_id/root_id/mention/quote — or with any of those cleared — would leave stale rows behind, corrupting recursive thread traversal. Fix: inside the same transaction, DELETE FROM edges WHERE from_id = ?1 AND kind IN ('reply','root','mention','quote') before the INSERT OR IGNORE inserts. The kind IN (...) filter lets future non-managed edge kinds coexist if they're ever added. Tests: reupsert_without_parent_drops_stale_edges (reply+root+mention -> 0 edges on top-level re-upsert), reupsert_replaces_mention_edges (M1 -> M2 swap leaves only the M2 edge). Added test-only probes in query.rs (test_only_count_edges_from / test_only_edge_target) reaching through Store::raw_conn() so the assertions don't need a public raw-SQL surface. * fix(threads-ingest): persist root post in ingest_thread Addresses Codex adversarial-review finding #3. run_ingest_thread used to paginate fetch_replies only. If the root post wasn't already in the store, `threads-cli ingest thread <id>` recorded a successful run while omitting the requested root entirely; for a thread with no replies it stored ZERO posts and still reported success. Fix: use Provider::fetch_thread, which drives the manifest's `post/conversation` edge and returns root + all descendants in one call. The orchestrator keeps dedup-via-HashSet, batched upserts, and the FetchRun start/end recording; only the fetch pattern changes. 3 new regression tests in ingest_tests.rs: - ingest_thread_persists_root_even_with_no_replies: 1 post stored, run.posts_fetched == 1, no error. - ingest_thread_persists_root_and_descendants: root + 2 replies -> 3 posts stored. - ingest_thread_empty_result_still_records_run_end: fetch_thread empty (root not found) still closes the run cleanly with 0 posts. MockProvider gains `with_thread(Vec<Post>)` so tests can seed fetch_thread output. * chore: codex adversarial review artifact + clippy nit - Add docs/reviews/codex-adversarial-2026-04-24.md so the review stays discoverable alongside the fixes it prompted. - Collapse nested if in init.rs per clippy::collapsible_if. After this commit, cargo test --workspace is 81 green; cargo clippy --workspace --all-targets is clean. * feat(ingest): `ingest engagement` crawls replies-to-me via BFS Adds the core "collect the reply tree under my posts/replies" workflow called out as a top PRD use-case. When somebody replies to one of your posts — and then someone replies to that reply, and a long reply forks into parent → child → grandchild siblings — those posts aren't reachable via `/me/threads` or `/me/replies`. This command fills the gap. Algorithm: - Seed = every post in the local store where `author_id = me.id` (`ingest me` must have run at least once). - BFS: for each seed, paginate `fetch_replies(seed_id)`; for each reply, recurse until `--depth` hops below the seed. Shared `HashSet<PostId>` dedupes across seeds and across levels so the same post is never fetched twice, even when two of your posts share a descendant. - Batched upserts (100 at a time) tagged with the run's `fetch_run_id` for provenance; FetchRun start/end recorded as usual. CLI surface: threads-cli ingest engagement [--depth N] Default depth is 8 (real Threads conversations rarely exceed 4–5). Uses `/{post-id}/replies` (top-level children + pagination) per Meta's docs — intentionally NOT `/{post-id}/conversation`, which returns the whole thread from root and wastes calls/storage on ancestors + siblings. Infra: - threads-store: `posts_by_author(UserId) -> Vec<PostId>` (direct SELECT). - StoreWrite: matching trait method + MockStore impl for tests. - 4 new orchestrator tests: direct replies, 3-level recursion, depth cap, cross-seed dedup. Total workspace tests: 81 → 85. --------- Co-authored-by: Chris George <chrisageorge@gmail.com>
1 parent d74499c commit 5a5a32d

38 files changed

Lines changed: 4254 additions & 55 deletions

.claude/settings.json

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,9 @@
3030
"Bash(sqlite3*)",
3131
"Bash(rm -rf target)",
3232
"Bash(touch*)",
33-
"Read(/Users/chris/projects/full-chaos/threads-cli/**)",
34-
"Write(/Users/chris/projects/full-chaos/threads-cli/**)",
35-
"Edit(/Users/chris/projects/full-chaos/threads-cli/**)"
33+
"Read(/Users/chris/projects/full-chaos/**)",
34+
"Write(/Users/chris/projects/full-chaos/**)",
35+
"Edit(/Users/chris/projects/full-chaos/**)"
3636
]
3737
}
3838
}

Cargo.lock

Lines changed: 38 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

crates/threads-cli/Cargo.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,17 @@ threads-store.workspace = true
1818
threads-ingest.workspace = true
1919

2020
anyhow.workspace = true
21+
chrono.workspace = true
2122
clap.workspace = true
23+
csv.workspace = true
2224
dirs.workspace = true
2325
serde.workspace = true
2426
serde_json.workspace = true
2527
toml.workspace = true
2628
tokio.workspace = true
2729
tracing.workspace = true
2830
tracing-subscriber.workspace = true
31+
url.workspace = true
32+
33+
[dev-dependencies]
34+
tempfile.workspace = true

crates/threads-cli/src/cli.rs

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
use std::path::PathBuf;
2+
3+
use clap::{ArgAction, Parser, Subcommand, ValueEnum};
4+
5+
#[derive(Debug, Parser)]
6+
#[command(
7+
name = "threads-cli",
8+
version,
9+
about = "Ingest, model, search, and export Threads content via the official Threads Graph API."
10+
)]
11+
pub struct Cli {
12+
/// Override the config-file path (default: ~/.config/threads-cli/config.toml).
13+
#[arg(long, global = true)]
14+
pub config: Option<PathBuf>,
15+
16+
/// Override the SQLite store path (default: ~/.local/share/threads-cli/store.db).
17+
#[arg(long, global = true)]
18+
pub db: Option<PathBuf>,
19+
20+
/// Increase logging verbosity (-v, -vv, -vvv).
21+
#[arg(short, long, action = ArgAction::Count, global = true)]
22+
pub verbose: u8,
23+
24+
/// Output format for commands that render records.
25+
#[arg(long, value_enum, default_value_t = OutputFormatArg::Human, global = true)]
26+
pub format: OutputFormatArg,
27+
28+
#[command(subcommand)]
29+
pub command: Command,
30+
}
31+
32+
#[derive(Copy, Clone, Debug, ValueEnum)]
33+
pub enum OutputFormatArg {
34+
Human,
35+
Json,
36+
Jsonl,
37+
Csv,
38+
}
39+
40+
impl From<OutputFormatArg> for crate::output::OutputFormat {
41+
fn from(v: OutputFormatArg) -> Self {
42+
match v {
43+
OutputFormatArg::Human => Self::Human,
44+
OutputFormatArg::Json => Self::Json,
45+
OutputFormatArg::Jsonl => Self::Jsonl,
46+
OutputFormatArg::Csv => Self::Csv,
47+
}
48+
}
49+
}
50+
51+
#[derive(Debug, Subcommand)]
52+
pub enum Command {
53+
/// Interactively register credentials for the Meta Threads app.
54+
Init(InitArgs),
55+
56+
/// Authentication subcommands.
57+
#[command(subcommand)]
58+
Auth(AuthCommand),
59+
60+
/// Ingest records from the provider into the local store.
61+
#[command(subcommand)]
62+
Ingest(IngestCommand),
63+
64+
/// Show a post, optionally the full thread rooted at it.
65+
Show(ShowArgs),
66+
67+
/// Full-text search the local store.
68+
Search(SearchArgs),
69+
70+
/// Export records from the store.
71+
Export(ExportArgs),
72+
}
73+
74+
#[derive(Debug, clap::Args)]
75+
pub struct InitArgs {
76+
/// Overwrite an existing config file.
77+
#[arg(long)]
78+
pub force: bool,
79+
}
80+
81+
#[derive(Debug, Subcommand)]
82+
pub enum AuthCommand {
83+
/// Run OAuth flow and store the access token.
84+
Login,
85+
/// Show the current token status.
86+
Status,
87+
/// Remove the stored token.
88+
Logout,
89+
}
90+
91+
#[derive(Debug, Subcommand)]
92+
pub enum IngestCommand {
93+
/// Ingest the authenticated user's threads + replies.
94+
Me,
95+
/// Ingest a single thread (root + descendants).
96+
Thread {
97+
/// The root post id.
98+
post_id: String,
99+
},
100+
/// BFS descend fetch_replies from every post you authored, up to
101+
/// `--depth` levels deep. Populates replies-to-your-replies (and their
102+
/// branching conversation trees) into the local store. Requires a prior
103+
/// `ingest me` so the store knows which posts you own.
104+
Engagement {
105+
/// Max BFS depth below each seed. Real Threads conversations
106+
/// rarely exceed 4-5 levels; 8 is a safe default.
107+
#[arg(long, default_value_t = 8)]
108+
depth: u32,
109+
},
110+
}
111+
112+
#[derive(Debug, clap::Args)]
113+
pub struct ShowArgs {
114+
/// The post id to show.
115+
pub post_id: String,
116+
/// Show the full thread rooted at this post (recursive CTE).
117+
#[arg(long)]
118+
pub thread: bool,
119+
}
120+
121+
#[derive(Debug, clap::Args)]
122+
pub struct SearchArgs {
123+
/// The FTS5 MATCH query.
124+
pub query: String,
125+
/// Limit the number of results.
126+
#[arg(long, default_value_t = 20)]
127+
pub limit: usize,
128+
}
129+
130+
#[derive(Debug, clap::Args)]
131+
pub struct ExportArgs {
132+
/// Write to a file instead of stdout.
133+
#[arg(long)]
134+
pub out: Option<PathBuf>,
135+
}
136+
137+
#[cfg(test)]
138+
mod tests {
139+
use super::*;
140+
use clap::CommandFactory;
141+
142+
#[test]
143+
fn cli_structure_is_valid() {
144+
Cli::command().debug_assert();
145+
}
146+
}

0 commit comments

Comments
 (0)