Skip to content

CLAUDE.md

Latest commit

 

History

History
165 lines (133 loc) · 33 KB

CLAUDE.md

File metadata and controls

165 lines (133 loc) · 33 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Commands

# Install dependencies
pip install -r requirements.txt

# Run all tests
pytest tests/ -v

# Run a single test file
pytest tests/test_scraper.py -v

# Run a single test
pytest tests/test_scraper.py::TestParseEntryList::test_kingregistration_columns -v

Architecture

The project is a pipeline that builds chess opponent dossiers from tournament entry lists. Each package is a self-contained pipeline stage:

scraper.py         → fetch entry list from tournament site → player names
pgnutil.py         → shared PGN text utility (split_pgn_games) used by every stage below
megabase/          → one-time SQLite index of ChessBase PGN export → game PGNs by name
lookup/            → Lichess + chess.com + web search → online profiles + game PGNs
  lichess.py       → search/profile/games/studies via the Lichess API
  chesscom.py      → username guessing + profile/games via the chess.com API
  websearch.py     → SearXNG JSON API client (general web search via a self-hosted instance, no account/key)
  broadcasts.py    → find Lichess broadcast (relay) games for a player via websearch.py,
                     since Lichess has no "search broadcasts by player" API
  uscf.py          → look up a tournament entrant's real FIDE nationality via their public
                     USCF record — a resolver confidence signal, not a game/profile source
analysis/          → PGN strings → opening repertoire + tendency stats
dossier/           → all of the above → rendered Markdown/HTML/JSON report
  report.py        → build_dossier() + render_markdown()/render_html()/render_json()
  db.py            → SQLite history of dossiers across repeated scans (separate from megabase —
                     this one is written to on every run, not built once and read-only)
pipeline/          → end-to-end orchestrator: tournament → dossier folder
  resolver.py      → name → (username, confidence, score, reasons) for Lichess and chess.com
  runner.py        → run_pipeline(): scrape → resolve → fetch (5 sources) → build → write
  cache.py         → always-on local SQLite cache: resolver results (TTL'd) + every broadcast
                     game ever found per player (never expires) — a performance cache, distinct
                     from dossier/db.py's opt-in cross-scan history

Data flow

  1. scraper.scrape_entry_list(tournament, site) → list[dict] of players with name, rating, section etc.
  2. megabase.query.get_player_games(name, db_path) → list[dict] each with a pgn key
  3. lookup.lichess.search(name) / lookup.chesscom.find_profile(name) → profile dicts; get_games() / games_as_pgn() / get_studies() + get_study_pgn() → PGN strings
  4. lookup.broadcasts.find_games(name, searxng_url) → PGN strings from Lichess broadcast rounds mentioning the player (optional, needs a SearXNG instance)
  5. analysis.openings.analyse_openings(pgn_strings, player) + analysis.stats.analyse_stats(pgn_strings, player) → dicts, each opening-line row carrying a capped list of the underlying games (with a URL when one exists)
  6. dossier.report.build_dossier(player, pgn_strings, profiles) → dossier dict; render_markdown() / render_html() / render_json() → string output

Key design decisions

  • All analysis functions are pure — they accept list[str] (PGN strings) and return dicts. No I/O. CLIs and dossier/report.py/pipeline/runner.py handle all sourcing.
  • Player name matching is token-based, not plain substring — analysis.openings._name_matches(player, header_name) requires every word in player to appear somewhere in header_name, so a tournament entry's truncated/compound surname (e.g. "Lagrave, Maxime" vs. a PGN's "Vachier-Lagrave, Maxime") still matches, while a same-surname different-person doesn't. megabase.query.get_player_games does the equivalent as ANDed SQL LIKE clauses, one per name token.
  • Tournament entry names get title-stripped before any matching — pipeline.resolver._strip_title removes a leading FIDE/USCF title ("GM Vachier-Lagrave, Maxime" → "Vachier-Lagrave, Maxime"); titles never appear in PGN headers or usernames and would otherwise poison every downstream match.
  • Never split multi-game PGN text with a \n(?=\[) regex — it splits between every header line, not between games, since a normal header block has no blank lines between tags. Always use pgnutil.split_pgn_games(), which round-trips through chess.pgn.read_game().
  • scraper.parse_entry_list requires at least one recognised column header from _HEADER_MAP before accepting a table, to skip nav/layout tables.
  • chess.com has no search API — lookup.chesscom.guess_usernames(name) generates candidates from Last, First / First Last patterns and find_profile() tries each until one resolves.
  • Lichess rate limiting — lookup.lichess sleeps 1s before game fetch requests. Its autocomplete endpoint is /api/player/autocomplete (not /api/users/autocomplete, which 404s) and 400s on a literal comma in the search term — search() strips it.
  • Lichess profile enrichment is opt-in per candidate — /api/player/autocomplete returns no rating/country/real-name data, only /api/user/{username} does, and only if the account owner filled it in. resolve_lichess fetches the full profile for just the top 2 name-ranked candidates to bound request volume.
  • megabase index is built once from a ChessBase PGN export (python -m megabase.indexer mega.pgn) and then queried read-only.
  • megabase name matching is whole-word, not substring, at the SQL level — megabase.query.get_player_games wraps White/Black in comma delimiters and matches LIKE '%,token,%'; a bare LIKE '%token%' matches a short token inside an unrelated word (e.g. "an" inside "Anderson"), which on an 11M-game database turns one token into millions of false positives, not a rare edge case. Passing rating additionally drops candidates whose matched side's Elo (read straight from the PGN's WhiteElo/BlackElo) is more than rating_tolerance points off — even whole-word matching can't disambiguate two different real people who share a common name.
  • Games with no public URL get a local interactive board — pipeline.runner._ensure_game_links collects them into one games-browser page per player at <output_dir>/games/<slug>/index.html (game list + click-to-load traversable board, pieces from python-chess's bundled Cburnett SVG set — the same one Lichess's default theme uses) and injects a GameURL header pointing at that game's anchor; a real GameURL/Link/Site URL already on the PGN (Lichess, chess.com) is left alone. The same viewer (_games_browser_html) also powers the high-confidence "recent games" pages (see below) and supports flipping the board and attaching a client-side engine.
  • Broadcast games never link straight out to the live broadcast page — pipeline.runner._delink_broadcast_games strips a broadcast game's GameURL/Site/Link/BroadcastURL headers before _ensure_game_links runs, so it's folded into the same local games-browser megabase games use instead of linking externally (analysis still sees the game either way — only the report's link target changes). The underlying PGNs are also persisted per-player in pipeline.cache's broadcast_games table (see below) rather than only living in that run's report.
  • The games-browser viewer runs a real UCI engine client-side — a Web Worker loaded from a CDN (Stockfish, an asm.js build, attached by default; a custom UCI-engine script URL is also accepted) drives an eval bar/number and a highlighted suggested move, updating as the user steps through the game. new Worker(crossOriginUrl) is rejected outright by browsers (SecurityError) regardless of CORS headers, so the worker is actually a same-origin Blob script that calls importScripts(url) — which does follow CORS. This only works for self-contained, single-file engine builds: one that fetches a separate .wasm file via a path relative to its own script location breaks, because that location is now the blob: URL, not the CDN directory (confirmed empirically against jsdelivr's WASM/NNUE Stockfish builds, which fail this way — the older asm.js build, with everything inlined, does not).
  • Lichess broadcasts can't be searched by player name — only by broadcast/tournament title (/api/broadcast/search) or by organizer username (/api/broadcast/by/{username}, not useful for a competitor). lookup.broadcasts.find_broadcast_round_ids works around this via a general web search (self-hosted SearXNG — see below) for <natural-order name> lichess.org (natural order, unquoted, bare domain — see the query-construction note further below), then fetches whatever round(s) it finds.
  • The pipeline cache (pipeline/cache.py) is always on, separate from the opt-in dossier_db — it exists purely to cut repeat-run cost, not to be queried by the user (that's what dossier/db.py is for). Two tables: resolutions caches each player's Lichess/chess.com resolver result (username/confidence/score/reasons) for --cache-ttl-days (default 7) — the guess/search sweep behind either resolver is the single most expensive, most-pointlessly-redone part of a re-run, since a player's identity rarely changes between scans; a cached "no match found" is saved too, since re-sweeping every chess.com guess for an unresolved player is the slowest case to redo, not the cheapest to skip. broadcast_games accumulates every broadcast PGN ever found per player, deduped by a hash of the PGN text, and never expires (a played game doesn't change) — this also raises effective recall against SearXNG's documented flakiness, since a game found on one run stays available even if a later run's search misses it. Default path <output_dir>/pipeline_cache.db; --no-cache disables both tables entirely (useful when iterating on resolver logic itself, where stale cached results would mask a code change).
  • Neither Lichess nor chess.com lets you search accounts by real name — confirmed directly: Lichess's /api/player/autocomplete?term=Magnus+Carlsen returns unrelated Magnus5/Magnus-/etc. accounts (pure username prefix-matching, real-name field never searched), and chess.com's own site-search endpoint (/callback/user/search, internally named web_user_callback_username_search) is also username-only and requires an authenticated session (401 without one). This is a hard platform limitation on both sides, not a gap in this codebase — a general web search is the only way to bridge "real name" → "unrelated username."
  • Web search is via a self-hosted SearXNG instance, not a hosted API — lookup.websearch.search(query, searxng_url) hits <searxng_url>/search?format=json. Chosen over a hosted option (e.g. Brave Search API) specifically to avoid an account-signup-plus-credit-card requirement (Brave requires a card even on its free tier, for identity verification) and per-query cost; the tradeoff is you host and maintain the instance yourself. JSON output must be enabled in the instance's settings.yml (search: formats: [html, json]) — it's off by default, including on public instances, for anti-abuse reasons, which is also why public instances can't just be pointed at directly (they 403/429 unauthenticated JSON requests in practice).
  • Name-search queries must be natural-order and unquoted, with a bare domain — lookup.lichess.find_usernames_via_search / lookup.chesscom.find_usernames_via_search reformat the entry-list "Last, First" name to natural "First Last" order (_natural_name_order) before searching, and build a query like Marek Antoni Kowalski chess.com — no exact-phrase quotes, no path fragment (chess.com/member, lichess.org/@). Confirmed empirically against a real unresolved case: exact-phrase-quoting the raw "Last, First" order finds nothing (real pages essentially never contain that literal comma-ordered substring), and separately, appending a path fragment as query text also kills recall on real search backends (it gets tokenized as unrelated keyword noise, e.g. "member", rather than treated as a URL/site hint) — either mistake alone is enough to silently miss an otherwise easily-findable profile page. A single self-hosted SearXNG instance backed by only 1–2 actually-responding engines (others get rate-limited/CAPTCHA'd quickly on a non-residential IP) is also inherently a bit flaky call-to-call — a miss is worth a retry before concluding the account isn't findable.
  • chess.com username guessing must drop middle names — lookup.chesscom.guess_usernames splits "Last, First Middle" on the comma, but without trimming to just the first given name, every guess embeds a literal space (e.g. "john derekheinichen") and 404s, silently breaking chess.com matching for anyone with a middle name on the entry list.
  • Resolver confidence factors in games played, not just name/rating/country — pipeline.resolver._composite_score adds an optional games-count signal (lookup.lichess/lookup.chesscom _slim_profile expose games_count); more games raises confidence, scaling up to _GAMES_FOR_FULL_SCORE (50). Separately, _confidence_for hard-caps confidence at "low" when games_count is known and below _MIN_GAMES_FOR_HIGH (5) — a same-name account with almost no games is too thin a sample to call "high confidence" even if name/rating/country line up perfectly, since that combination is just as consistent with a different person. chess.com's _slim_profile also now exposes real_name (the account's real-name field, distinct from display_name which falls back to the username) — a genuine real-name match is direct evidence and can upgrade a late, generic-looking username guess to high confidence, the same way Lichess's real_name already could.
  • _name_score's surname-substring floor must respect word boundaries when the candidate is a real name, not a username — a real production false positive: entrant "Imran, Sheikh Waali" scored a 0.65 name-match floor against total stranger "Simran Kolagad"'s chess.com real_name, because the substring check stripped all whitespace before comparing ("simran kolagad" → "simrankolagad", which contains "imran" as a raw substring even though the names are unrelated). Usernames genuinely need the old substring-anywhere check ("jsmith" for "Smith, John" has no delimiter to preserve), but a real_name/display name with a space in it has genuine word boundaries — _name_score now only applies the floor there when the surname matches a whole word of the candidate, the same whole-word-vs-substring principle already applied to megabase queries above. Only saved from surfacing as a false "high confidence" match that time because the account also happened to be 18+ years stale (see the recency cap below) — a differently-timed collision wouldn't have been. Covered by TestNameScore in tests/test_pipeline.py.
  • A catastrophic rating mismatch is a second hard confidence cap, not just a scoring penalty — a real production case exposed this: a generic guess ("john") landed on an unrelated stranger's real chess.com account, and because that account happened to have thousands of games and a "preferred" country, the composite score nearly reached "high" despite an ~1000-point rating gap. _composite_score now also returns rating_ok (False only when a rating comparison was actually made and came back fully clamped to the 0.0 floor — i.e. the gap is at or past _RATING_TOLERANCE), and _confidence_for caps confidence at "low" whenever rating_ok is False, the same way it does for a too-small games_count. No rating data at all leaves rating_ok True — absence of evidence isn't evidence of a mismatch.
  • Account recency is a third signal and hard cap, distinct from games_count — a lot of games proves the account is a real, active identity, but says nothing about whether it's still this player's active account: hundreds of games from 5+ years ago is weak evidence for a tournament entrant playing today. lookup.lichess/lookup.chesscom _slim_profile now expose last_active (an ISO date, from Lichess's seenAt or chess.com's last_online — both already present on the same profile call, no extra request) — _recency_score scales it from 1.0 (active within _RECENCY_FULL_SCORE_DAYS, 90 days) down to 0.0 (inactive _RECENCY_ZERO_SCORE_DAYS, ~2 years, or more), and _composite_score returns a third hard-cap flag recency_ok (False only when last_active is known and older than _STALE_ACCOUNT_DAYS, 5 years) that _confidence_for treats the same way as rating_ok/too-few-games. No last_active data leaves recency_ok True.
  • chess.com guessing scores every guess and keeps the best, not just the first hit — previously _resolve_chesscom_by_guessing stopped at the first guess that resolved to a real profile, so a weak, coincidental match early in the guess list (e.g. a bare first name colliding with an unrelated stranger) could block a later, better-evidenced guess from ever being considered. Now it behaves like the Lichess candidate-ranking path: try every guess, score each hit, keep the best.
  • The country signal cross-checks actual FIDE nationality instead of assuming every entrant is US-based — lookup.uscf.get_fide_country(uscf_id) scrapes the public USCF MSA member page (https://www.uschess.org/msa/MbrDtlMain.php?<uscf_id>, keyed by the uscf_id already scraped from the entry list — no name-based lookup or disambiguation needed) for the entrant's "FIDE Country" field, and converts the FIDE/IOC-style 3-letter federation code to ISO alpha-2 via _FIDE_TO_ISO2 for comparison against a candidate's Lichess/chess.com country. pipeline.runner.run_pipeline looks this up once per player and passes it to both resolvers as fide_country; _country_score uses it when available and only falls back to the old blanket "US-preferred" heuristic when it isn't (most club-level players have no FIDE record at all). This was a real, demonstrated bug, not a hypothetical: the blanket US bias caused a correct non-US candidate with a 0.90 real-name match to lose to a wrong US candidate with only a 0.65 match, purely because of nationality — confirmed fixed against live data (score flipped from 0.75-wrong/0.64-correct to 0.53-wrong/0.93-correct once the actual FIDE nationality was known).

Supported tournament sites

Site --site flag URL pattern
kingregistration.com kingregistration (default) /entrylist/<id>
chessaction.com chessaction /tournaments/advance_entry_list.php?tid=<id>

Full URLs are auto-detected; --site is only needed for ID shorthands.

Step 6 pipeline details

pipeline/resolver.py:

  • _strip_title(name) — strips a leading FIDE/USCF title
  • resolve_lichess(name, rating=None, searxng_url=None, fide_country=None) / resolve_chesscom(name, rating=None, searxng_url=None, fide_country=None) → (username, "high"|"low"|None, score, reasons) — each candidate is scored on name/handle similarity (weight 0.5), rating closeness to rating when available (weight 0.3, tighter tolerance if it's a FIDE rating rather than an online blitz/rapid one), account country (weight 0.2 — see below), games played (weight 0.2, scaling up to _GAMES_FOR_FULL_SCORE), and account recency (weight 0.15, scaling from _RECENCY_FULL_SCORE_DAYS down to _RECENCY_ZERO_SCORE_DAYS — see below); missing signals are dropped from the weighted average rather than penalising the candidate. score >= 0.55 → high, >= 0.30 → low, else rejected (None) — except _confidence_for overrides a would-be "high" down to "low" when the account has fewer than _MIN_GAMES_FOR_HIGH games on record, when the rating comparison came back catastrophically bad (rating_ok is False), or when the account's been inactive for _STALE_ACCOUNT_DAYS+ (recency_ok is False), regardless of score (see the games-count, rating-mismatch, and recency design notes above).
    • Lichess: scores all search() candidates (cheap name-only pass first, then fetches full profiles for just the top 2)
    • chess.com: scores every guess that resolves to a real profile and keeps the best, on guess specificity ("firstlast" vs. a bare "first" stands in for name similarity, since every guess is mechanically derived from the name) unless the account has a genuine real_name on file, in which case an actual name-similarity score is used instead when it's higher
    • Both: if searxng_url is given and the above didn't already reach high confidence, also try find_usernames_via_search() (SearXNG search for the player's name plus lichess.org or chess.com) and keep whichever candidate scores best — catches a personalized handle with no relation to the player's name (e.g. Magnus Carlsen's real Lichess account is the pseudonymous DrNykterstein), findable only via the account's linked real name, which neither Lichess's username-only autocomplete nor any mechanical chess.com guess would ever surface. This path needs a running SearXNG instance ($SEARXNG_URL / --searxng-url) — without one, personalized handles like this are simply not discoverable; see the SearXNG setup note below.

pipeline/runner.py:

  • run_pipeline(tournament, ...) — full orchestration; returns list[Path] of written files
  • Per player, pulls games from up to 5 sources in order: megabase, Lichess games, Lichess studies, chess.com games, Lichess broadcasts (only if searxng_url is set)
  • Writes <output_dir>/<slug>.{html,md,json} per player and a combined file in html/markdown mode
  • exclude filters the scraped player list by substring before the loop (e.g. to skip your own entry)
  • match_score/match_reasons from the resolver get injected into each profile dict alongside confidence, and rendered next to it in the report
  • _ensure_game_links() runs on the collected PGNs (html/markdown modes only) before build_dossier(), generating the per-player games-browser page for anything without a public URL
  • Recent-games viewer for high-confidence matches — when a Lichess or chess.com match is "high" confidence, _recent_games_page() builds a dedicated openingtree-style games-browser page (<output_dir>/games/<slug>/recent_lichess.html / recent_chesscom.html) restricted to games from the last calendar year (_RECENT_GAMES_DAYS), fetched independently of the smaller pool used for opening analysis (_fetch_lichess_recent_games uses Lichess's since filter directly; chess.com always pulls a dedicated 12-month archive regardless of --chesscom-months). The profile dict gets recent_games_url/recent_games_count, rendered as a link next to that profile in the report. Low-confidence matches deliberately skip this — their game history may belong to someone else.
  • Optional cross-scan database — pass dossier_db=<path> (CLI: --dossier-db) to also save every dossier via dossier.db.save_dossier(), building a queryable history across repeated scans (see below). Off by default; failures are logged and don't abort the run.
  • Always-on resolver/broadcast cache — cache_db/--cache-db (default <output_dir>/pipeline_cache.db), no_cache/--no-cache to disable, cache_ttl_days/--cache-ttl-days (default 7) for how long a cached resolution stays valid; see pipeline/cache.py above. A cached resolution/broadcast search is logged with a [cached] suffix so it's obvious from the run output which players skipped a live lookup.

dossier/db.py (opt-in, off by default):

  • SQLite schema: one scans row per (tournament, day) run_pipeline() call, one dossiers row per player per scan (stats/openings/profiles stored as JSON columns)
  • save_dossier(db_path, tournament, site, dossier) — upserts; rerunning the same tournament the same day updates that day's row instead of duplicating, but a later rescan (or a different tournament) starts a new scan and keeps history
  • player_history(db_path, player) / latest_dossier(db_path, player) — matched by name slug (pipeline.runner._slug-equivalent), so formatting differences in how the name is written don't create phantom separate players
  • CLI: python -m dossier.db --db dossiers.db history "Smith, John" / ... scans [--tournament NAME]

Roadmap

  • Steps 1–6 are complete, including MegaDatabase integration.
  • Also done: game-count-aware and rating-mismatch-aware match confidence, chess.com guessing scores every candidate instead of stopping at the first hit, web search migrated from Brave to self-hosted SearXNG, a client-side engine (Stockfish by default, off until toggled, top-3 multi-PV lines) with flip/eval/best-move in an enlarged/centered games viewer, a recent-games (last 12 months) viewer for high-confidence matches, and an opt-in cross-scan dossier database.
  • Verified against real data across two real tournament scans: with the games-count and rating-mismatch hard caps in place, a wrong-but-plausible candidate (large game count, matching country, but a rating far outside tolerance) correctly stays at low confidence instead of being fooled by the incidental signals; separately, fixing the search query construction (natural name order, unquoted, bare domain — see above) took a personalized-handle case that the guess-only path could never reach from "not discoverable at all" to "found, high confidence" once SearXNG was live, though a single self-hosted instance with only 1–2 reliably-responding engines means recall on any one query isn't 100% consistent call to call.
  • Remaining: combined PDF output. A SearXNG instance still needs a card-free signup-free way to run reliably 24/7 outside a sandbox with no Docker (the one used for development here was run from source in a venv — see the setup note below) — and even with search enabled, it depends on which upstream engines happen to be responding, so this stays a best-effort discovery path, not a guarantee.
  • Fixed 2026-07-10, both confirmed against real players in the recurring WQ260708 tournament: lookup.broadcasts.find_broadcast_round_ids had never picked up the natural-order/unquoted/bare-domain query fix already applied to lichess.py/chesscom.py's search (it was still exact-phrase-quoting the raw "Last, First" order and appending a /broadcast path fragment), silently missing real, findable rounds — and a chess.com generic single-word guess (e.g. "john") could collide with an unrelated stranger's real account and score "high" purely from country/games/recency when no rating was on file to catch it, locking out the search path entirely before it ever ran. See _MIN_NAME_SCORE_FOR_HIGH/name_ok in pipeline/resolver.py.
  • Also done 2026-07-10: broadcast games no longer link out to the live Lichess broadcast page (folded into the local games-browser instead) and are persisted per-player so a later run doesn't lose one SearXNG happens to miss; and the resolver/broadcast-search cache described in the caching roadmap item below is implemented (pipeline/cache.py) — the first, highest-impact slice of that item. Still open from that same item: megabase-query caching and any change to the deliberate 1s Lichess rate-limit sleeps themselves.
  • Investigated 2026-07-14 — user-reported "drop in quality" in generated reports, specifically wrong/low-quality profile matches: audited the live dossiers/WQ260708/ output (33 players, a 2026-07-11 scan) rather than guessing, and found _name_score's surname-substring floor (see the design-decision note above) producing false-positive name matches against unrelated strangers' real names once word boundaries were erased by space-stripping — fixed. The four confidence hard caps (games/rating/recency/name_ok) added 2026-07-10 are working as designed and are not the regression — most of what they're capping down (accounts 8–19 years inactive) is correctly-suspect evidence, not a bug; _name_score was the one genuinely broken input feeding them.

Future roadmap ideas (not yet started)

  • Mistake/theory-deviation insights per opponent — flag where an opponent's games diverge from known theory or contain outright blunders, not just which openings they play. Two different signals, both new: (1) "deviates from theory" doesn't need an external opening book — the megabase itself already has 11.7M games, so "theory" at any position can be derived from what the megabase's strongest/most-common continuations actually are, compared ply-by-ply against the opponent's games (analysis/openings.py already builds a move-tree per opponent; this would need the same tree built from megabase as a baseline to diff against). (2) "mistakes" needs actual eval, which today only exists client-side (the games-browser's Stockfish Web Worker, browser-only, nothing persisted) — a report-time version would need a server-side UCI engine pass (e.g. the stockfish PyPI binary wrapped via python-chess) over each opponent's games, which is a real new dependency and meaningfully slower per-dossier build, so probably opt-in (--analyse-blunders or similar) rather than default.
  • Feed your own games against an opponent into their dossier — right now build_dossier() only ever sees the opponent's own games from the 5 sourcing paths; there's no way to inject your games against them specifically. Would need: a way to supply your own PGN pool (a file, or your own Lichess/chess.com username reused through the existing lookup.lichess/lookup.chesscom fetchers), a head-to-head filter matching the opponent's name on the other side of the board from you, and a distinct "vs. you" section in the report (and, per the "combined reports and databases" ask, a way to keep accumulating this across scans rather than starting over each run — dossier/db.py's existing cross-scan history is the natural place this could plug in, since it's already keyed by opponent name-slug).
  • Personal repertoire vs. opponent tendencies — let the user define their own opening repertoire (as White and Black) and have the report highlight which of the opponent's games actually entered one of those lines, i.e. "here's what they did the last N times they faced what you actually play," rather than their opening stats in the abstract. Representationally this is close to what analysis/openings.py already builds (a move-sequence tree per player) — the new part is a second tree (the user's own repertoire, probably authored as a small PGN of main lines) and a matching pass that walks each opponent game against it to find where the game and the repertoire diverge, then surfaces just those games instead of (or in addition to) the full opening breakdown.
  • Cloud hosting for dossiers/history, accessible from multiple devices — today everything is local-only: HTML/MD/JSON dossier files on disk, dossier/db.py's SQLite history, and the 16GB megabase.db. The megabase itself is impractical to host remotely at that size for a personal tool (and read-only/build-once anyway, so it doesn't need to move), but the actual per-scan outputs (dossier files + the history db) are small and are the part worth syncing — e.g. pushing rendered HTML to a static host (S3/R2/GitHub Pages on a private repo) or standing up a minimal read-only web view backed by dossier/db.py. Needs a real answer on access control before anything ships, since dossiers currently contain other real players' names/data with no auth model at all today.
  • Caching / faster pipeline runs — partially done 2026-07-10. A full WQ260708-sized scan (~33 players) takes on the order of 10+ minutes end to end. The resolver-result cache and the broadcast-games cache described here are now implemented — see pipeline/cache.py and the design-decision/roadmap notes above — so a re-run within --cache-ttl-days (default 7) skips the expensive chess.com guess sweep / Lichess autocomplete / SearXNG search entirely for a player already resolved, and previously-found broadcast games are never lost even if a later SearXNG search misses them. Still open: the two 1s-per-request rate-limit sleeps (lookup.lichess._RATE_DELAY, pipeline.runner's own time.sleep(1.0) calls around megabase/profile fetches) are deliberate rate-limit courtesy, not accidental, so not just removable — reducing them would need either a smarter shared rate-limiter (track actual time-since-last-request instead of always sleeping the full second) or accepting more 429s; and megabase SQL queries aren't cached at all yet (untested whether they're actually a meaningful chunk of the 10+ minutes, or dwarfed by the network calls — worth profiling a real run before adding cache machinery there).

SearXNG setup

Self-host an instance to enable --searxng-url (personalized-handle discovery + Lichess broadcast search):

docker run -d --name searxng -p 8080:8080 \
  -v "$(pwd)/searxng-config:/etc/searxng" \
  searxng/searxng

On first run, it writes a default searxng-config/settings.yml — edit it to add:

search:
  formats:
    - html
    - json

...then restart the container (docker restart searxng). Verify with:

curl "http://localhost:8080/search?q=test&format=json"

Point the pipeline at it with --searxng-url http://localhost:8080 or $SEARXNG_URL.