Step 1: scrape tournament entry lists (kingregistration + chessaction)

* Add Playwright scraper for kingregistration.com entry lists

Scrapes tournament pages at /entrylist/<id>, waits for JS rendering,
extracts player table rows, and normalises column headers to canonical
field names (name, rating, uscf_id, section, club, etc.).

Outputs CSV or JSON to stdout; --save-html dumps rendered page for debugging.

https://claude.ai/code/session_01VQfqug9MDEyRFydkmES4n2

* Simplify scraper to requests+BS4, add README

Site is static HTML so Playwright is unnecessary. Replaced it with
requests + BeautifulSoup — no browser install required.

Added README with install instructions, usage examples, output format
docs, and a roadmap for the dossier-building steps ahead.

https://claude.ai/code/session_01VQfqug9MDEyRFydkmES4n2

* Add chessaction.com support, extend README

scraper.py now handles both kingregistration.com and chessaction.com.
Site is auto-detected from full URLs; --site flag selects it for
ID shorthands. Header normalisation extended to cover chessaction
column variants (pre-rating, player name, uscf#, team, etc.).

README updated with a supported-sites table, per-site usage examples,
and a flags reference table.

https://claude.ai/code/session_01VQfqug9MDEyRFydkmES4n2

* Add tests; fix nav-table false-positive in parser

25 offline unit tests covering resolve_url, _detect_site, _normalize,
and parse_entry_list with HTML fixtures for both sites and edge cases
(empty table, blank rows, no table, multiple tables).

Fixed parser to require at least one recognised chess column header
before accepting a table, so nav/layout tables are skipped correctly.

https://claude.ai/code/session_01VQfqug9MDEyRFydkmES4n2

* Add .gitignore

Excludes __pycache__, bytecode, build artifacts, and common output
files (csv, json, html) from version control.

https://claude.ai/code/session_01VQfqug9MDEyRFydkmES4n2

* Add GitHub Actions CI workflow

Runs pytest on every push and pull request using Python 3.11.

https://claude.ai/code/session_01VQfqug9MDEyRFydkmES4n2

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: jlhe97

.github/workflows/ci.yml

@@ -0,0 +1,20 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+
+      - run: pip install -r requirements.txt
+
+      - run: pytest tests/ -v

.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+*.csv
+*.json
+*.html

README.md

@@ -0,0 +1,107 @@
+# Chess Dossier Builder
+
+Build opponent dossiers for players registered in the same chess tournament.
+
+## Step 1 — Scrape tournament entry lists
+
+`scraper.py` fetches a tournament entry list and returns the registered players as CSV or JSON.
+
+**Supported sites**
+| Site | URL pattern |
+|---|---|
+| [kingregistration.com](https://www.kingregistration.com) | `/entrylist/<id>` |
+| [chessaction.com](https://chessaction.com) | `/tournaments/advance_entry_list.php?tid=<id>` |
+
+### Install
+
+```bash
+pip install -r requirements.txt
+```
+
+### Tests
+
+```bash
+pytest tests/ -v
+```
+
+All tests run offline using HTML fixtures — no network required.
+
+### Usage
+
+**kingregistration.com** (default)
+```bash
+python scraper.py Challenge34
+python scraper.py Challenge34 --output json
+python scraper.py https://www.kingregistration.com/entrylist/Challenge34
+```
+
+**chessaction.com**
+```bash
+python scraper.py nKGioA== --site chessaction
+python scraper.py "https://chessaction.com/tournaments/advance_entry_list.php?tid=nKGioA=="
+```
+
+When a full URL is passed, `--site` is auto-detected and can be omitted.
+
+**All flags**
+```
+python scraper.py <tournament> [--site kingregistration|chessaction]
+                               [--output csv|json]
+                               [--save-html FILE]
+```
+
+| Flag | Default | Description |
+|---|---|---|
+| `--site` | `kingregistration` | Site to use for ID shorthands |
+| `--output` | `csv` | Output format: `csv` or `json` |
+| `--save-html FILE` | — | Save the raw HTML for debugging |
+
+### Output
+
+**CSV (default)**
+```
+"name","rating","uscf_id","section","club","state"
+"Smith, John","1850","12345678","Open","Metro Chess Club","NY"
+```
+
+**JSON**
+```json
+[
+  {
+    "name": "Smith, John",
+    "rating": "1850",
+    "uscf_id": "12345678",
+    "section": "Open",
+    "club": "Metro Chess Club",
+    "state": "NY"
+  }
+]
+```
+
+Column headers are normalised automatically across both sites
+(e.g. `"Rtng"`, `"Pre-Rating"`, `"USCF Rating"` all map to `"rating"`).
+Unknown headers are passed through lowercased.
+
+### Piping output
+
+```bash
+python scraper.py Challenge34 > entries.csv
+python scraper.py Challenge34 --output json | jq '.[].name'
+```
+
+### Debugging an unknown layout
+
+If the scraper prints `No player table found`, run with `--save-html` and
+inspect the HTML to identify the right selector to add:
+
+```bash
+python scraper.py Challenge34 --save-html page.html
+```
+
+## Roadmap
+
+- [x] Step 1 — Scrape tournament entry lists (kingregistration, chessaction)
+- [ ] Step 2 — Look up each player on USCF / chess.com / Lichess
+- [ ] Step 3 — Fetch recent games per player
+- [ ] Step 4 — Analyse openings and tendencies
+- [ ] Step 5 — Generate per-opponent dossier report

requirements.txt

@@ -0,0 +1,3 @@
+requests>=2.31.0
+beautifulsoup4>=4.12.0
+pytest>=8.0.0

scraper.py

@@ -0,0 +1,205 @@
+"""
+Scrape tournament entry lists from kingregistration.com or chessaction.com.
+
+Usage:
+  python scraper.py <tournament> [--site kingregistration|chessaction] [--output csv|json]
+
+Tournament can be:
+  - A tournament ID shorthand  (resolved using --site, default: kingregistration)
+  - A full URL                 (site auto-detected; --site flag ignored)
+
+Examples:
+  python scraper.py Challenge34
+  python scraper.py Challenge34 --output json
+  python scraper.py nKGioA== --site chessaction
+  python scraper.py https://www.kingregistration.com/entrylist/Challenge34
+  python scraper.py "https://chessaction.com/tournaments/advance_entry_list.php?tid=nKGioA=="
+  python scraper.py Challenge34 --save-html page.html
+"""
+
+import sys
+import json
+import argparse
+
+import requests
+from bs4 import BeautifulSoup
+
+
+# --- Site URL templates ---------------------------------------------------
+
+_SITES = {
+    "kingregistration": "https://www.kingregistration.com/entrylist/{tid}",
+    "chessaction":      "https://chessaction.com/tournaments/advance_entry_list.php?tid={tid}",
+}
+
+_HEADERS = {
+    "User-Agent": (
+        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
+    ),
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.9",
+}
+
+# Normalise common column header variants to canonical field names
+_HEADER_MAP = {
+    "name": "name",
+    "player": "name",
+    "player name": "name",
+    "full name": "name",
+    "last, first": "name",
+    "rating": "rating",
+    "rtng": "rating",
+    "uscf rating": "rating",
+    "pre-rating": "rating",
+    "pre rating": "rating",
+    "uscf": "uscf_id",
+    "uscf id": "uscf_id",
+    "uscf#": "uscf_id",
+    "id": "uscf_id",
+    "section": "section",
+    "division": "section",
+    "club": "club",
+    "team": "club",
+    "state": "state",
+    "grade": "grade",
+    "school": "school",
+    "city": "city",
+}
+
+
+# --- URL resolution -------------------------------------------------------
+
+def _detect_site(url: str) -> str | None:
+    for site in _SITES:
+        if site in url:
+            return site
+    return None
+
+
+def resolve_url(tournament: str, site: str = "kingregistration") -> str:
+    if tournament.startswith("http"):
+        return tournament
+    tid = tournament.rstrip("/").split("/")[-1]
+    return _SITES[site].format(tid=tid)
+
+
+# --- Fetching & parsing ---------------------------------------------------
+
+def fetch_html(url: str) -> str:
+    resp = requests.get(url, headers=_HEADERS, timeout=15)
+    resp.raise_for_status()
+    return resp.text
+
+
+def parse_entry_list(html: str) -> list[dict]:
+    soup = BeautifulSoup(html, "html.parser")
+
+    for table in soup.find_all("table"):
+        rows = table.find_all("tr")
+        if len(rows) < 2:
+            continue
+
+        headers = [th.get_text(strip=True) for th in rows[0].find_all(["th", "td"])]
+        if not headers:
+            continue
+
+        # Skip tables with no recognised chess columns (e.g. nav/layout tables)
+        known = {_HEADER_MAP.get(h.lower().strip()) for h in headers} - {None}
+        if not known:
+            continue
+
+        players = []
+        for row in rows[1:]:
+            cells = [td.get_text(strip=True) for td in row.find_all(["th", "td"])]
+            if not cells or all(c == "" for c in cells):
+                continue
+            players.append(_normalize(dict(zip(headers, cells))))
+
+        if players:
+            return players
+
+    print("No player table found. Page text preview:", file=sys.stderr)
+    print(soup.get_text(separator="\n", strip=True)[:2000], file=sys.stderr)
+    return []
+
+
+def _normalize(entry: dict) -> dict:
+    return {
+        _HEADER_MAP.get(k.lower().strip(), k.lower().strip()): v
+        for k, v in entry.items()
+    }
+
+
+# --- Main entry point -----------------------------------------------------
+
+def scrape_entry_list(tournament: str, site: str = "kingregistration",
+                      save_html: str | None = None) -> list[dict]:
+    # Auto-detect site from full URLs so --site flag is optional
+    if tournament.startswith("http"):
+        detected = _detect_site(tournament)
+        if detected:
+            site = detected
+
+    url = resolve_url(tournament, site)
+    print(f"[{site}] Fetching: {url}", file=sys.stderr)
+
+    html = fetch_html(url)
+
+    if save_html:
+        with open(save_html, "w", encoding="utf-8") as fh:
+            fh.write(html)
+        print(f"HTML saved to {save_html}", file=sys.stderr)
+
+    players = parse_entry_list(html)
+    print(f"Found {len(players)} player(s).", file=sys.stderr)
+    return players
+
+
+# --- Output helpers -------------------------------------------------------
+
+def output_csv(players: list[dict]) -> None:
+    if not players:
+        print("No data.")
+        return
+    headers = list(players[0].keys())
+    print(",".join(f'"{h}"' for h in headers))
+    for player in players:
+        print(",".join(f'"{player.get(h, "")}"' for h in headers))
+
+
+def output_json(players: list[dict]) -> None:
+    print(json.dumps(players, indent=2, ensure_ascii=False))
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Scrape a chess tournament entry list.",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=__doc__,
+    )
+    parser.add_argument("tournament", help="Tournament ID shorthand or full URL")
+    parser.add_argument(
+        "--site", choices=list(_SITES), default="kingregistration",
+        help="Site to scrape (default: kingregistration). Ignored when a full URL is given.",
+    )
+    parser.add_argument(
+        "--output", choices=["csv", "json"], default="csv",
+        help="Output format (default: csv)",
+    )
+    parser.add_argument(
+        "--save-html", metavar="FILE",
+        help="Save raw HTML to FILE for debugging",
+    )
+    args = parser.parse_args()
+
+    players = scrape_entry_list(args.tournament, site=args.site, save_html=args.save_html)
+
+    if args.output == "json":
+        output_json(players)
+    else:
+        output_csv(players)
+
+
+if __name__ == "__main__":
+    main()