Skip to content

AGENTS.md

Latest commit

 

History

History
271 lines (250 loc) · 20 KB

AGENTS.md

File metadata and controls

271 lines (250 loc) · 20 KB
Raw

AGENTS.md — SourceCapsule

X (Twitter) Article/post → one self-contained, offline .html file (all media + quoted tweets inlined). Shipped as a Tampermonkey/Violentmonkey userscript.

Before any operational work (releases, the share Worker, hosted anything), read SOURCE_OF_TRUTH.md — it declares which environment, branch, and artifact version is authoritative. If it doesn't exist yet, create it from the current facts and confirm them with the owner.

Stack & layout

  • Plain JavaScript, zero-build. The shipped artifact is sourcecapsule.user.js — the file you edit is the file users install. No bundler.
  • package.json exists only for dev tooling (ESLint + Prettier) and CI. Node 18+.
  • test/smoke.mjs — Node test of the DOM-free engine. examples/sample-export.html is generated by it (gitignored from Prettier).

Commands

npm install          # dev tooling only (eslint, prettier)
npm test             # smoke-test the assembly/inlining engine (no browser)
npm run lint         # eslint .
npm run format:check # prettier --check .  (npm run format to fix)

CI (.github/workflows/lint.yml) runs lint + format check + npm test on push/PR to main. There is no automated browser test — real X testing is manual (see CONTRIBUTING.md).

Architecture (5 lines)

  1. Fragile layer: reads X's DOM → a plain-object model. ALL X selectors live in the CONFIG block at the top of sourcecapsule.user.js.
  2. Passive capture layer: X's own web app fetches TweetDetail / TweetResultByRestId GraphQL responses to render the page; a MAIN-world bridge tees those bodies to handleNetworkCapturePayload, which harvests full long-form ("note") text into capturedNoteTweets and <parentId, quotedId, quotedHandle> triples into capturedQuotedRefs. Quote-only responses are included (the bridge body filter must always contain quoted_status), bodies are capped at 6 MB with explicit truncation diagnostics, and full-body hashing prevents distinct same-envelope responses being deduplicated. Free source of truth — no extra network call, and it survives whatever the DOM later virtualizes away.
  3. Syndication layer: each embedded/quoted tweet is re-fetched by id from cdn.syndication.twimg.com/tweet-result (with retry + backoff; 404 stops early) and its quote card is rebuilt from that authoritative data (enrichQuotesViaSyndication → syndicationToQuoteBlock). Runs AFTER capture-based recovery so most quote permalinks are already resolved and no round-trip is needed. Threads get a per-post media/link diff (enrichThreadViaSyndication); focused single posts get the same diff via enrichFocusedPostViaSyndication (both share applySyndicationDataToSegment), so a lazily-lost image that never produced a DOM block is still recovered. Successful payloads are cached per export (syndicationSuccessCache; failures never cached), and replies prepend their parent post via enrichReplyContextViaSyndication (pref-gated).
  4. Ship-blocker layer (strict export): assessExportCompleteness(model) walks the fully-recovered model and returns any dead-end the reader would see (missing quote permalinks, uncaptured quote content, failed images, videos with no bytes or poster). When strict mode is on (default) and blockers remain, repairExportBlockers() first retries each through the layer that owns it (captured refs → pool permalink matcher → syndication quote rebuild → media rescue) after a short pause; only if the export is STILL incomplete is the download blocked with a confirm modal offering Retry recovery (re-runs the repair round in place), a buildDiagnosticBundle() "Copy diagnostic" button, and Ship-anyway/Cancel.
  5. Stable layer: GM_xmlhttpRequest fetch (bypasses CORS) → base64 → assembleHtml() / renderLlmMarkdown() → download. Touches only the model, never the DOM. The model is the contract, so X's frequent DOM churn rarely hits the engine.

Export modes

  • The default click on every control is share — "Create AI link". It is the trigger's own action, so share is deliberately NOT a menu item. On a focused post the trigger passes includeThread from postControlCaptureMode, re-read at click time, so a thread becomes ONE capsule (## Full Thread, Post n of N) rather than a link per post. share-thread is the AI-link twin of library-thread: same escape hatch, first in the focused-post menu, for when X's virtualization hides the follow-ups at button-render time.
  • Menu offers Create AI link (full thread) and Save full thread (focused posts only), then Save to library, Save with note / tags, Copy clean Markdown. The engine still supports library-share, both, html and md; they are just no longer menu items.
  • Plain share publishes at the default 7-day expiry with no modal — the labelled click IS the confirmation. Only the combined save-and-share flow prompts for an expiry.
  • postControlCaptureMode returns includeThread: isFocusedPost, not isThread. X paints a status page with the root post ALONE and fetches the conversation a beat later, so isThread is false for the first second — and a click in that window used to publish a one-post "thread". isThread still drives the tooltip; the scope does not depend on it.
  • waitForConversation runs before the media scroll on any focused-post thread export. Without it forceLoadMedia has nothing to scroll in that same window, returns at once, and the model is built from the root post by itself. It exits as soon as more than one top-level post has been mounted and held still, so the 6s ceiling is only ever paid in full by a reply-less post.
  • copyText bounds navigator.clipboard.writeText. Chromium can leave that promise pending forever when the window is not OS-focused; unbounded, it stranded the export AFTER the capsule was published — button stuck on "Exporting...", sticky toast lying about a link that existed.
  • placePostControl re-places an overlay control once X's header caret mounts. On a focused post the caret renders after the article, so the control lands as an absolute overlay on top of the post's own text and used to stay there for the life of the page.
  • Save to library (saveToLibrary) writes each export into a per-post folder under a root the user picks once: <root>/<date>/<handle>-<id>/{<handle>-<id>.html?, <handle>-<id>.llm.md, media/}. It uses the File System Access API (getRootDir persists the FileSystemDirectoryHandle in IndexedDB; the browser may re-confirm write permission ~once per session). Chromium only; non-Chromium falls back to a single .zip of the same per-post tree via the built-in store-only writer (buildZip/crc32). The media/ files are images + video poster stills only — raw video bytes are never bundled (an LLM cannot watch video), so the bundle stays small.
  • The .llm.md is honest about itself: a ## What This File Is header states it is text + metadata; in bundle mode it references the real media/... files (collectBundleMediaFiles → pathById → renderLlmMarkdown(..., { mediaFiles })); it never names a file that is not on disk.
  • Four prefs (layout date|flat, contents full|lean, strictExport bool, replyContext bool) live in localStorage. Toggled via userscript-manager menu commands (registerSettingsMenu) OR the MV3 popup — no in-app panel. strictExport defaults on; when on, the ship-blocker layer runs before assembly. replyContext defaults on; when on, a reply export prepends the replied-to post as a labelled context card (enrichReplyContextViaSyndication; honest tombstone note when the parent is gone on X, skipped when the parent is already a captured thread post).
  • Status-page quick save captures the visible same-author thread by default. It progressively scrolls from the top and keeps cloned tweet nodes so X virtualization does not erase earlier posts. Capture remains best-effort and is labelled as such. Every focused-post button also exposes Save full thread in its drop-down (postControlCaptureMode returns THREAD_EXPORT_TYPES whenever isFocusedPost=true, regardless of isThread), as an escape hatch when X's virtualization or a false-positive thread-boundary heading defeats auto-detection at button-render time.
  • Share with AI uploads only after confirmation to the configured share Worker, with a default 7-day expiry (1/30 days optional), a 25 MB cap, and no raw video. The Cloudflare Worker + R2 implementation lives in share-worker/. The create call sends the post's canonical permalink so that expiry leaves a tombstone, not a dead end: content is deleted at expiry but a ~300-byte record survives and /c/<id>(.md) answers 410 with a page linking back to the original X post. The Worker re-validates sourceUrl against a canonical x.com/<handle>/status/<id> pattern — it must never emit a URL the client merely asserted, or the endpoint becomes an open redirect. Tombstones are hard-deleted 180 days after expiry, or immediately on DELETE. The scheduled sweep must keep reading expiresAt from customMetadata on the list page; reverting to a per-capsule GET makes cleanup cost grow without bound as tombstones accumulate.
  • npm run build:extension generates an experimental MV3 package in dist/sourcecapsule-extension/ using the same userscript source plus a thin GM compatibility layer.

Reply archive (experimental, branch work)

Separate from post export: capture the replies to a post — full text, author, timestamp, parent id, and media links — not just an inventory of reply ids.

  • Menu (focused posts): Capture replies (experimental) · Latest / Top / Relevant runs a scroll pass over that X surface; Download reply archive (Markdown + CSV) writes the deliverables.
  • Sources, in order of authority: passive GraphQL bodies (full note_tweet text, in_reply_to_status_id_str, extended_entities media) > DOM scrape > per-id syndication for gaps. networkCapturePatterns().body must keep matching conversation_id_str — a page of plain text replies matches none of the media/note/quote terms, and dropping those bodies silently starves the archive. Keep extension-src/page-bridge.js aligned.
  • The merge invariant: mergeReplyArchiveRecords lets content only accumulate. A populated field is never overwritten by an empty one and the longer text wins, so a truncated timeline preview can never clobber full note text and a reply that vanishes from a later surface keeps what was already captured. Do not "simplify" this into a plain object spread — that is exactly the bug that made the earlier iteration an id inventory.
  • Storage is IndexedDB (createReplyArchiveStore), with a localStorage fallback. A ~1,000-reply archive with full text overruns localStorage's ~5 MB origin quota, and that failure is silent, so every write returns an explicit storageError that the probe surfaces as its own sticky error toast.
  • Media is links only, on purpose. Downloading media for thousands of replies is out of scope; the archive records image/video/poster URLs and lets the reader follow them.
  • Gap recovery (enrichReplyArchiveViaSyndication) runs after the first archive write, capped at 250 ids / concurrency 3. A 404 is authoritative and becomes an honest tombstone; other failures are returned in errors, never swallowed.
  • Coverage is best effort and the Markdown receipt says so, listing known-but- uncaptured reply ids. X's public reply counter is a reference, never a denominator.

Unattended capture (experimental, Windows)

node scripts/sourcecapsule-capture.mjs --url "<post>" --json captures a post and prints the finished AI readable link as JSON, with no clicks, prompts, or clipboard use.

  • Direction matters. The browser spawns native messaging hosts; nothing outside can dial into the browser. So the service worker holds a long-lived port to native-host/sourcecapsule-host.mjs, the host owns the Windows named pipe \\.\pipe\sourcecapsule-capture, and the CLI connects to that pipe. One host owns the pipe, which is also what enforces one capture at a time. SOURCECAPSULE_PIPE overrides the name so tests never fight the host a running browser already owns.
  • The registered host must be a real .exe. Chromium is unreliable about launching .bat/.cmd native hosts. scripts/install-native-host.ps1 compiles native-host/launcher.cs with the .NET compiler already on Windows, installs everything to %LOCALAPPDATA%\SourceCapsule\native-host, and registers it under HKCU only. Two traps it handles: PowerShell 5.1 writes a UTF-8 BOM that Chromium rejects outright, and a running host holds its own exe open so a reinstall must stop it first.
  • The extension ID is pinned by a key in extension-src/manifest.json, because the host manifest has to name a fixed chrome-extension:// origin. Do not regenerate it.
  • Capture reuses runExport('share'). runAutomatedShareCapture only waits for the page and the passive capture layer, then hands off. Do not add a second parsing path here.
  • The tab is ACTIVE in an UNFOCUSED window, never hidden. A hidden tab has requestAnimationFrame paused and timers throttled, which starves forceLoadMedia and manufactures strict-mode blockers.
  • The browser must run with --disable-features=CalculateNativeWinOcclusion. Unfocused is not enough: Windows occlusion tracking treats a fully covered window exactly like a hidden tab, so the capture window fetched X's TweetDetail and then never mounted it. The capsule came back holding the root post alone. Measured on one thread: without the flag the conversation wait settles on 1 top-level post after 6.9s; with it, 13 posts in 1.7s and a capsule holding all 8. start-sourcecapsule-browser.ps1 passes it and writes it into the shortcut, and -Status exits 3 when the running browser lacks it.
  • An unattended capture reports its thread scope. The result carries capturedPosts and the conversation counters waitForConversation recorded, and warns when thread scope was requested and one post came back. A capsule that silently drops seven posts is the failure mode that made this bug survive a live verification.
  • Strict mode stays strict. With no human to answer confirmShipDespiteIncomplete, a surviving blocker raises NeedsOwnerError and the CLI reports needs_owner with the assessment counts. Never "fix" an unattended failure by relaxing the gate. runExport's catch rethrows when automation is true: absorbing the error into a toast nobody can read turned a strict refusal into no_capsule with no blockers. test/strict-capture.test.mjs is the regression fixture - it drives the real controller boundary against media and syndication layers that answer 404 (authoritative in both, so no recovery can succeed).
  • stdout is a contract. formatResult allowlists the published fields so transport details (the correlation id) can never leak into what another program parses.
  • If the bridge reports Specified native messaging host not found, the usual cause is that the extension is not actually running, not a bad registration. Launching the browser with --load-extension=dist/sourcecapsule-extension is the reliable fix.
  • --load-extension does not survive a restart, by design. It installs at Chromium's COMMAND_LINE location (the profile's Secure Preferences records "location": 8), and a start without the flag does not load it. So an ordinary browser restart has no extension at all - the worker is not asleep, it does not exist - and the CLI cannot reach the host. scripts/start-sourcecapsule-browser.ps1 is the durable path: -InstallShortcut -InstallStartup for the owner-visible launcher, -Status to diagnose (nonzero when the running browser lacks the flag), -Restart -Verify to repair and confirm. Launching the browser again while it is already running cannot fix it: the second launch hands its arguments to the existing process and the flag is dropped. A one-time Load unpacked records UNPACKED instead and needs no launcher, but it is a manual UI step and cannot coexist with the command-line copy (same key, same ID).
  • The host must not outlive its browser. launcher.cs closes the Node child's stdin once the browser's stream ends, and the host exits on stdin end/close/error or a failed stdout write. Without both, a killed browser left an orphan holding \\.\pipe\sourcecapsule-capture, and every later CLI request was answered by a host whose extension was gone - a full-timeout hang that reads exactly like "the host is unreachable".

Gotchas

  • CORS is the whole reason it's a userscript. Media bytes from pbs.twimg.com / video.twimg.com can only be read via GM_xmlhttpRequest with the @connect grants. The library settings menu also needs @grant GM_registerMenuCommand / GM_unregisterMenuCommand.
  • Selectors break periodically when X reships its markup → fix in CONFIG, keep old selectors as fallbacks. Misses log [SourceCapsule] … warnings, never crash.
  • Quoted-post tombstones are honest notes, not blockers. [data-testid="tombstone"] where a quote card would be means the quoted post is gone on X itself (banned/deleted/ restricted). Captured as a quote-tombstone block that renders an explicit note plus a receipt/manifest count (quoteTombstones); never counted missing and never gates the export — do not "fix" that by making it a blocker.
  • Video preservation is best-effort. Full MP4s inline when X exposes them (no size cap, so archives can be large); HLS-only/blocked/failed videos fall back to poster + source link and are recorded as incomplete media — never counted as "captured." Long-form ("note") posts return only a preview from syndication; the network-capture layer passively grabs the full text from GraphQL responses X's web app already fetched (noteTweetsFromCapturedBody → capturedNoteTweets map, swapped in during enrichThreadViaSyndication / recoverQuoteNoteText, marked with a note-recovered block). Posts whose full text was never delivered to the browser stay flagged truncated.
  • Embedded-tweet media comes from syndication, not the DOM. The DOM only supplies each quote's position + status id; enrichQuotesViaSyndication overwrites its text/media. If syndication is blocked/changes, set CONFIG.useSyndication=false to fall back to DOM scraping (which has the known duplicate/misattribution issues). The token is derived from the id (syndicationToken, same scheme as react-tweet).
  • Syndication must augment, not erase, DOM-only semantics. Public tweet-result payloads do not reliably carry polls or rich link-card metadata. Every quote replacement goes through mergeQuoteAfterSyndication, which preserves those blocks recursively. A parent payload with a quoted_tweet also creates the quote when the DOM card never mounted.
  • Never guess multi-video ownership by array order. Network candidates are bitrate-sorted, not post-sorted. Poster/media keys are authoritative; a keyless fallback is allowed only for one unresolved video and one logical media ID.
  • Markdown is Prettier-ignored on purpose (hand-formatted). The engine is exposed to Node via a typeof module guard at the bottom of the userscript — keep it.
  • Repo: wolfgang-aura/SourceCapsule (public/OSS). @downloadURL/@updateURL point at main — keep the default branch named main.

Out of scope (v1)

Batch/bookmark export, in-app settings panel (manager menu commands only — see Export modes), HLS reassembly, OCR/transcripts, AI-generated summaries/media descriptions, hosted dashboard, accounts/billing/permanent-link quotas, Chrome Web Store publication, guaranteed complete capture of arbitrarily long threads, and active fetching of long-form (“note”) post text (recovery is passive-only: the full text is used when X already delivered it to the browser; otherwise the preview stays flagged truncated).