v1.4.0: HTTP 413 auto-bisect for the 15 MB upload cap

HiroAlleyCat · HiroAlleyCat · commit c0e3c03351c5 · 2026-06-05T07:17:03.000-04:00
LOCOSP rolled out a temporary 15 MB body cap on every wdgwars.pl upload
endpoint on 2026-06-05 (workaround for CloudLinux LVE killing the PHP
worker mid-buffer on bodies above ~20 MB). The server returns a structured
413 envelope with max_bytes + received instead of a generic 500 maintenance
page. Cap is expected to be removed in ~2 weeks after a host migration.

This release reacts to that envelope automatically so the standalone CLI
path (no --chunk-size, single POST) survives an over-cap upload without
operator intervention. The proactive --chunk-size 10000 default in the
--schedule path was already comfortably under the cap and is unchanged.

Added
- HTTP 413 auto-bisect in _upload_chunks. On a payload-too-large response
  the offending chunk is split in half (row-count, header preserved on
  both halves) and both halves are pushed back onto the work queue.
  Recursion bottoms out cleanly when a single-row chunk still 413s
  (recorded as a failure, other chunks continue).
- _halve_chunk(bytes) helper; semantics mirror _split_bytes.
- tests/test_413_autosplit.py: 7 tests covering the halver, single-bisect
  success, double-bisect when first halves still oversize, one-row-still-
  413 failure recording, and forward-compat for a future max_bytes bump
  (no hard-coded 15 MB constant — the client reads it off the envelope).

Changed
- _upload_chunks reworked from a fixed-list for-loop to a deque so the
  bisect path can push retries back onto the front of the queue without
  breaking the iteration counter. 200 / 429 / non-JSON paths byte-for-
  byte identical to v1.3.0.
- --chunk-size help text now describes the auto-bisect safety net.
- README error table picks up the 413 row alongside the existing 524.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,46 @@ All notable changes to wigle-to-wdgwars are documented here. Format
 follows [Keep a Changelog](https://keepachangelog.com/) and the
 project uses [Semantic Versioning](https://semver.org/).
 
+## [1.4.0] - 2026-06-05 - 15 MB upload cap (HTTP 413 auto-bisect)
+
+LOCOSP rolled out a temporary 15 MB body cap on every wdgwars.pl upload
+endpoint on 2026-06-05. The cap is a workaround for the CloudLinux LVE
+on the shared host killing the PHP worker mid-buffer on bodies above
+roughly 20 MB. Until the planned host migration lands (~2 weeks), the
+server returns a structured 413 envelope with `max_bytes` + `received`
+instead of a generic 500.
+
+This release reacts to that envelope automatically. The proactive
+`--chunk-size` flag is unchanged and remains the recommended path for
+scheduled cron uploads, but standalone CLI runs are now resilient too.
+
+### Added
+
+- HTTP 413 auto-bisect. When a chunk comes back with
+  `{error: payload-too-large, max_bytes, received}`, the offending
+  chunk is split in half (row-count, header preserved on both halves)
+  and both halves are pushed back onto the work queue. Recursion
+  bottoms out cleanly when a chunk is one row and still 413 (recorded
+  as a failure, other chunks continue).
+- `_halve_chunk(bytes) -> tuple[bytes, bytes] | None` helper. Public
+  enough for sibling tools (Muninn, Heimdall) to vendor if they want
+  the same behavior; semantics match `_split_bytes`.
+- `tests/test_413_autosplit.py`: 7 tests covering the halver, single-
+  bisect success, double-bisect when first halves still oversize,
+  one-row-still-413 failure recording, and forward-compat for a
+  future `max_bytes` change (no hard-coded 15 MB constant).
+
+### Changed
+
+- `_upload_chunks` reworked from a fixed-list `for` loop to a
+  `collections.deque` so the bisect path can push retries back onto
+  the front of the queue without breaking the iteration counter.
+  Behavior on 200/429/non-JSON paths is byte-for-byte identical to
+  v1.3.0.
+- `--chunk-size --help` text now describes the auto-bisect behavior
+  so the flag's role is clear: proactive splitting to avoid the
+  Cloudflare 524 timeout window, with the 413 path as a safety net.
+
 ## [1.3.0] - 2026-06-03 - Family-parity catch-up
 
 Closes the flag-surface gaps between wigle-to-wdgwars and its sibling
diff --git a/README.md b/README.md
@@ -716,6 +716,7 @@ server treats it as a valid file.
 | 400 | `{"error":"Invalid data format"}` | Most likely you POSTed a CSV to `/api/upload` (no `-csv` suffix). Wrong endpoint, not a malformed file. |
 | 401 | `{"error":"Invalid API key"}` | Bad/expired key, or you used `Authorization: Bearer …` instead of `X-API-Key:`. |
 | 429 | `{"error":"Another upload is already being processed …","retry_after":N}` | Per-account queue. Wait `retry_after` seconds. |
+| 413 | `{"error":"payload-too-large","max_bytes":15728640,"received":N,...}` | Body exceeded the 15 MB hosting cap LOCOSP added 2026-06-05. The client auto-bisects the offending chunk and retries both halves — no flag needed. |
 | 524 | (HTML from Cloudflare) | Origin timed out. Chunk smaller. Rows are still ingesting on the origin. |
 
 ### WiGLE API (the `--from-wigle` pull side)
diff --git a/tests/test_413_autosplit.py b/tests/test_413_autosplit.py
@@ -0,0 +1,183 @@
+"""Tests for the HTTP 413 auto-bisect path introduced in v1.4.0.
+
+Background: 2026-06-05 LOCOSP rolled out a temporary 15 MB body cap on every
+wdgwars.pl upload endpoint with a structured 413 envelope
+(`{error: payload-too-large, max_bytes, received, ...}`). The client now
+reacts to that envelope by halving the offending chunk and retrying both
+halves; this module exercises that behavior end-to-end without touching the
+network.
+"""
+from __future__ import annotations
+
+import io
+import json
+import unittest
+from unittest import mock
+
+import wigle_to_wdgwars as w2w
+
+
+HEADER = (
+    b"WigleWifi-1.6,appRelease=2.74,model=Pixel\n"
+    b"MAC,SSID,AuthMode,FirstSeen,Channel,RSSI,CurrentLatitude,"
+    b"CurrentLongitude,AltitudeMeters,AccuracyMeters,Type\n"
+)
+
+
+def _row(mac_suffix: int) -> bytes:
+    return (
+        f"aa:bb:cc:00:00:{mac_suffix:02x},Net{mac_suffix},[WPA2],"
+        f"2026-06-05 10:00:00,6,-65,41.46,-82.18,200,5,WIFI\n"
+    ).encode()
+
+
+def _csv_with_rows(n: int) -> bytes:
+    return HEADER + b"".join(_row(i) for i in range(n))
+
+
+def _413_envelope(received: int, max_bytes: int = 15_728_640) -> str:
+    return json.dumps(
+        {
+            "ok": False,
+            "error": "payload-too-large",
+            "http_status": 413,
+            "max_bytes": max_bytes,
+            "received": received,
+            "message": "Your upload is too large for this hosting plan...",
+            "retry_after": 0,
+        }
+    )
+
+
+def _ok_envelope(imported: int = 1, total: int = 1) -> str:
+    return json.dumps(
+        {
+            "ok": True,
+            "imported": imported,
+            "captured": imported,
+            "updated": 0,
+            "duplicates": 0,
+            "no_gps": 0,
+            "bad_rows": 0,
+            "total": total,
+        }
+    )
+
+
+class HalveChunkTests(unittest.TestCase):
+    def test_halve_preserves_header_on_both_halves(self) -> None:
+        csv = _csv_with_rows(8)
+        halves = w2w._halve_chunk(csv)
+        assert halves is not None
+        left, right = halves
+        # Both halves carry the 2-line WigleWifi-1.6 + column header.
+        self.assertTrue(left.startswith(HEADER))
+        self.assertTrue(right.startswith(HEADER))
+        # Roughly even split (8 rows → 4+4).
+        left_rows = [ln for ln in left.decode().splitlines()[2:] if ln]
+        right_rows = [ln for ln in right.decode().splitlines()[2:] if ln]
+        self.assertEqual(len(left_rows), 4)
+        self.assertEqual(len(right_rows), 4)
+
+    def test_halve_returns_none_on_one_row(self) -> None:
+        csv = _csv_with_rows(1)
+        self.assertIsNone(w2w._halve_chunk(csv))
+
+    def test_halve_returns_none_on_empty(self) -> None:
+        self.assertIsNone(w2w._halve_chunk(HEADER))
+
+
+class UploadChunks413Tests(unittest.TestCase):
+    """End-to-end: _upload_chunks should bisect-and-retry on a 413 envelope."""
+
+    def _run_with_responses(self, chunks: list[bytes], responses: list[tuple]) -> tuple[int, list[tuple]]:
+        """Drive _upload_chunks with a queued list of (status, body) tuples.
+        Returns (rc, list_of_post_call_arg_summaries).
+        """
+        post_calls: list[tuple] = []
+        response_iter = iter(responses)
+
+        def fake_post(body, name, key, field):
+            status, raw = next(response_iter)
+            post_calls.append((status, len(body)))
+            return status, raw, 0.01
+
+        with mock.patch.object(w2w, "_post_one", side_effect=fake_post), \
+             mock.patch.object(w2w, "_hwm_record"), \
+             mock.patch("time.sleep"), \
+             mock.patch("sys.stderr", new=io.StringIO()), \
+             mock.patch("sys.stdout", new=io.StringIO()):
+            rc = w2w._upload_chunks(
+                chunks, name="t.csv", key="k", field="file",
+                dry_run=False, cooldown_sec=0.0,
+            )
+        return rc, post_calls
+
+    def test_413_bisects_and_both_halves_succeed(self) -> None:
+        csv = _csv_with_rows(8)
+        responses = [
+            (413, _413_envelope(received=20_000_000)),
+            (200, _ok_envelope(imported=4)),
+            (200, _ok_envelope(imported=4)),
+        ]
+        rc, calls = self._run_with_responses([csv], responses)
+        self.assertEqual(rc, 0)
+        # 1 initial POST + 2 halves
+        self.assertEqual(len(calls), 3)
+        self.assertEqual(calls[0][0], 413)
+        self.assertEqual(calls[1][0], 200)
+        self.assertEqual(calls[2][0], 200)
+        # Halves are strictly smaller than the original chunk.
+        self.assertLess(calls[1][1], calls[0][1])
+        self.assertLess(calls[2][1], calls[0][1])
+
+    def test_413_bisects_twice_when_first_halves_still_oversize(self) -> None:
+        csv = _csv_with_rows(8)
+        responses = [
+            (413, _413_envelope(received=40_000_000)),
+            (413, _413_envelope(received=20_000_000)),
+            (200, _ok_envelope(imported=2)),
+            (200, _ok_envelope(imported=2)),
+            (200, _ok_envelope(imported=4)),
+        ]
+        rc, calls = self._run_with_responses([csv], responses)
+        self.assertEqual(rc, 0)
+        # Initial + left-half-413 + 2 quarter halves + right half = 5 POSTs.
+        self.assertEqual(len(calls), 5)
+        statuses = [s for s, _ in calls]
+        self.assertEqual(statuses, [413, 413, 200, 200, 200])
+
+    def test_413_single_row_chunk_records_failure_and_continues(self) -> None:
+        """If a chunk is already one row and still gets 413, record it as a
+        failure and move on. Other chunks must still process."""
+        one_row = _csv_with_rows(1)
+        ok_chunk = _csv_with_rows(2)
+        responses = [
+            (413, _413_envelope(received=16_000_000)),
+            (200, _ok_envelope(imported=2)),
+        ]
+        rc, calls = self._run_with_responses([one_row, ok_chunk], responses)
+        # Aggregate ok=False because one payload failed.
+        self.assertEqual(rc, 1)
+        self.assertEqual(len(calls), 2)
+        self.assertEqual(calls[0][0], 413)
+        self.assertEqual(calls[1][0], 200)
+
+    def test_413_max_bytes_not_hard_coded_against_server_value(self) -> None:
+        """If LOCOSP later raises the cap to 25 MB, the client must still
+        react to the same envelope shape — not a baked-in constant.
+        We exercise this by sending a 413 envelope with max_bytes=25 MB and
+        confirming the bisect path fires identically."""
+        csv = _csv_with_rows(8)
+        responses = [
+            (413, _413_envelope(received=30_000_000, max_bytes=26_214_400)),
+            (200, _ok_envelope(imported=4)),
+            (200, _ok_envelope(imported=4)),
+        ]
+        rc, calls = self._run_with_responses([csv], responses)
+        self.assertEqual(rc, 0)
+        self.assertEqual(len(calls), 3)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/wigle_to_wdgwars.py b/wigle_to_wdgwars.py
@@ -34,11 +34,12 @@
 """
 from __future__ import annotations
 
-__version__ = "1.3.0"
+__version__ = "1.4.0"
 GITHUB_REPO = "HiroAlleyCat/wigle-to-wdgwars"
 GITHUB_URL = f"https://github.com/{GITHUB_REPO}"
 
 import argparse
+import collections
 import gzip
 import json
 import logging
@@ -790,6 +791,27 @@ def _split_csv(csv_path: Path, chunk_rows: int) -> list[bytes]:
     return _split_bytes(_read_csv_bytes(csv_path), chunk_rows)
 
 
+PAYLOAD_TOO_LARGE_ERROR = "payload-too-large"
+
+
+def _halve_chunk(chunk_bytes: bytes) -> tuple[bytes, bytes] | None:
+    """Bisect a WiGLE CSV chunk into two row-count halves, header preserved
+    on each. Returns None if the chunk has fewer than 2 data rows (cannot
+    bisect further). Used to react to LOCOSP's 15 MB upload cap (2026-06-05):
+    on HTTP 413 the offending chunk is halved and both halves are retried.
+    """
+    raw = chunk_bytes.decode("utf-8").splitlines(keepends=False)
+    if len(raw) < 4:
+        return None
+    h1, h2, *data_rows = raw
+    mid = len(data_rows) // 2
+    if mid < 1:
+        return None
+    left = (h1 + "\n" + h2 + "\n" + "\n".join(data_rows[:mid]) + "\n").encode("utf-8")
+    right = (h1 + "\n" + h2 + "\n" + "\n".join(data_rows[mid:]) + "\n").encode("utf-8")
+    return left, right
+
+
 def _aggregate(payloads: list[dict]) -> dict:
     """Merge per-chunk response envelopes into one summary."""
     keys = ("imported", "captured", "updated", "duplicates", "no_gps", "bad_rows", "merged_samples")
@@ -813,7 +835,14 @@ def _aggregate(payloads: list[dict]) -> dict:
 
 def _upload_chunks(chunks: list[bytes], name: str, key: str, field: str,
                    dry_run: bool, cooldown_sec: float) -> int:
-    """POST pre-split CSV chunks to WDGoWars. Returns shell exit code (0 ok)."""
+    """POST pre-split CSV chunks to WDGoWars. Returns shell exit code (0 ok).
+
+    Resilient to HTTP 413 from LOCOSP's 15 MB upload cap (2026-06-05): any
+    chunk that comes back with `{error: payload-too-large, max_bytes, received}`
+    is bisected and both halves are pushed back onto the work queue. Recursion
+    bottoms out when a chunk is one row and still 413 (recorded as a failure,
+    other chunks continue).
+    """
     total_kb = sum(len(c) for c in chunks) / 1024
     print(
         f"[wdgwars] POST {ENDPOINT} field={field} file={name} "
@@ -823,18 +852,50 @@ def _upload_chunks(chunks: list[bytes], name: str, key: str, field: str,
     if dry_run:
         print("[wdgwars] dry-run: not sending", file=sys.stderr)
         return 0
+    queue: collections.deque[bytes] = collections.deque(chunks)
     payloads: list[dict] = []
-    for idx, body in enumerate(chunks, 1):
+    attempt = 0
+    splits = 0
+    while queue:
+        attempt += 1
+        body = queue.popleft()
         try:
             status, raw, dur = _post_one(body, name, key, field)
         except urllib.error.URLError as e:
-            sys.exit(f"[wdgwars] network error on chunk {idx}/{len(chunks)}: {e}")
+            sys.exit(f"[wdgwars] network error on attempt {attempt}: {e}")
         try:
             data = json.loads(raw)
         except json.JSONDecodeError:
             data = {"ok": False, "error": "non-json response", "raw": raw[:300]}
+
+        if status == 413 and data.get("error") == PAYLOAD_TOO_LARGE_ERROR:
+            halves = _halve_chunk(body)
+            max_b = data.get("max_bytes")
+            recv = data.get("received")
+            if halves is None:
+                print(
+                    f"[wdgwars] attempt {attempt} HTTP 413 with 1 row, cannot "
+                    f"bisect further (max_bytes={max_b} received={recv}). "
+                    f"Recording as failure, continuing.",
+                    file=sys.stderr,
+                )
+                payloads.append(data)
+                continue
+            left, right = halves
+            queue.appendleft(right)
+            queue.appendleft(left)
+            splits += 1
+            print(
+                f"[wdgwars] attempt {attempt} HTTP 413 "
+                f"(max_bytes={max_b} received={recv}); bisecting and retrying "
+                f"({len(left) // 1024}+{len(right) // 1024} KB halves)",
+                file=sys.stderr,
+            )
+            time.sleep(cooldown_sec)
+            continue
+
         print(
-            f"[wdgwars] chunk {idx}/{len(chunks)} HTTP {status} in {dur:.1f}s "
+            f"[wdgwars] attempt {attempt} HTTP {status} in {dur:.1f}s "
             f"imported={data.get('imported')} dup={data.get('duplicates')} "
             f"merged={data.get('merged_samples')} bad={data.get('bad_rows')}",
             file=sys.stderr,
@@ -847,9 +908,14 @@ def _upload_chunks(chunks: list[bytes], name: str, key: str, field: str,
             print(f"[wdgwars] 429 cooldown, sleeping {wait:.0f}s", file=sys.stderr)
             _cooldown_record(wait)
             time.sleep(wait)
-        elif idx < len(chunks):
+        elif queue:
             time.sleep(cooldown_sec)
-    if len(chunks) == 1:
+    if splits:
+        print(
+            f"[wdgwars] auto-split {splits} chunk(s) on 413 over the run",
+            file=sys.stderr,
+        )
+    if len(payloads) == 1:
         print(json.dumps(payloads[0]))
         return 0 if payloads[0].get("ok") else 1
     agg = _aggregate(payloads)
@@ -1574,8 +1640,11 @@ def main() -> int:
     ap.add_argument("--no-version-check", action="store_true",
                     help="skip the daily GitHub release check entirely")
     ap.add_argument("--chunk-size", type=int, default=0,
-                    help="split CSV into N-row chunks to dodge Cloudflare 524s (0=single POST). "
-                         "10000 is a safe default for large uploads.")
+                    help="proactively split CSV into N-row chunks (0=single POST). "
+                         "10000 is a safe default for large uploads. The tool also "
+                         "reacts to LOCOSP's 15 MB upload cap (HTTP 413 envelope) "
+                         "by bisecting any over-cap chunk automatically, so this "
+                         "flag is mostly belt-and-suspenders.")
     ap.add_argument("--chunk-cooldown", type=float, default=5.0,
                     help="seconds to sleep between chunks (default: 5)")
     ap.add_argument("--whoami", action="store_true",
