Skip to content

Releases: Yggdrasil-AI-labs/wigle-to-wdgwars

v1.6.0 — skip already-processed uploads

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 15 Jun 19:58

Stops the daily --from-wigle pull from re-downloading uploads it already pushed. Records each successfully-pushed transid in ~/.config/wigle-to-wdgwars/processed-transids.json and skips re-pulling it (WiGLE regenerates each CSV server-side, minutes per large upload). Recorded only on a real non-dry-run success; new --reprocess flag forces a re-pull. +7 tests.

v1.5.1 — survive slow WiGLE CSV exports

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 15 Jun 19:37

Fix: wigle_download_csv retries the CSV download with a longer read timeout (600s, then 900s) instead of a flat 300s. WiGLE builds the CSV server-side, so large uploads can take minutes; the 300s ceiling crashed the daily run with an uncaught TimeoutError.

v1.5.0 — trailing-window gate (--since)

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 15 Jun 19:37

Add a --since gate (default 7d) so cron jobs stop re-uploading the entire WiGLE history every tick. Row-level FirstSeen filter before chunking; --all-time disables it.

v1.4.0 — HTTP 413 auto-bisect

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 15 Jun 19:37

Auto-bisect a chunk that trips the WDGoWars upload cap: on HTTP 413, split the batch and retry the halves instead of failing the run.

v1.3.0 — family-parity catch-up

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 03 Jun 16:01
a5f904e

Summary

Closes the flag-surface gaps between wigle-to-wdgwars and its sibling feeders Muninn and Heimdall. Each of the five flags below already shipped in the other two uploaders; this brings wigle-to-wdgwars to full parity.

Flag Muninn Heimdall Wigle before Wigle after
--update yes yes no yes
--preview yes yes no yes
-q / --quiet yes yes no yes
--no-version-check yes yes no yes
--api-url yes yes no yes
--watch yes n/a n/a n/a (documented)

What changed

Added

  • --update: git pull --ff-only when this is a git checkout, otherwise raw-GitHub fetch of wigle_to_wdgwars.py + requirements.txt. Atomic replace, pip refresh after. Lifted from Heimdall.
  • --preview: prints first 6 WiGLE CSV rows as JSON-lines, decompresses gzip transparently. No network. Matches Heimdall + Muninn semantics.
  • -q / --quiet: suppress informational banners (errors still print). Gates the version-check nudge.
  • --no-version-check: skip the daily GitHub releases probe entirely.
  • --api-url URL: override the CSV upload endpoint. Useful for staging or local mocks. Aircraft JSON uploads still use the signed endpoint unchanged.
  • Daily-cached version-check nudge in main(), gated by both flags above.
  • SECURITY.md, refreshed for the v1.3.0 surface.

Not added

  • --watch is intentionally absent. WiGLE CSV uploads are bulk one-shot; the polling equivalent for this tool is --from-wigle --schedule, which already exists. Documented in CHANGELOG.

Tests

  • tests/test_family_parity.py: 10 new offline tests covering preview_csv (gzip, short files, empty data sections), _check_for_update cache hits + network-error path, --api-url runtime ENDPOINT override, and --update / --preview dispatch in main(). Network mocked throughout.
  • Existing 46 tests still pass. Total: 56 tests, all green.

Test plan

  • AST parse + module import
  • --version returns 1.3.0
  • --help shows all 5 new flags
  • --preview /tmp/sample.csv prints JSON rows, no upload
  • --api-url http://localhost:9999/fake --dry-run swaps ENDPOINT, doesn't POST
  • python -m unittest discover tests/ — 56 pass
  • bash scripts/smoke.sh — all green, including the dry-run systemd unit render with no key leak
  • Live --schedule install against a real systemd user manager (deferred to pre-release manual checklist)
  • Live --update against this branch once merged (verifies the raw-GitHub path)

v1.2.1 — family-alignment housekeeping

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 03 Jun 04:30
179f788

What's Changed

  • v1.2.1: family-alignment housekeeping (gungnir pin + safety nets) by @HiroAlleyCat in #5

Full Changelog: v1.2.0...v1.2.1

v1.2.0 — guided setup + auto-installed daily timer

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 02 Jun 17:03
b574a44

From a fresh clone to a working dry-run daily timer in one command:

git clone https://github.com/HiroAlleyCat/wigle-to-wdgwars.git
cd wigle-to-wdgwars
./setup.sh                    # Linux / Mac / Pi — saves both keys, installs daily timer
REM Windows
setup.bat

What's new

  • --setup interactive wizard. Prompts for the WDGoWars API key, validates against /api/me, saves to ~/.config/wigle-to-wdgwars/wdgwars.key (mode 600). Then the WiGLE "Encoded for use" token, validated against /api/v2/file/transactions. Then offers a daily timer.
  • --save-key / --save-wigle-key for non-interactive provisioning.
  • --schedule / --unschedule auto-installer. Writes the right artifact for your host: systemd user .timer+.service (Linux+systemd), cron line (Mac / Linux-without-systemd), schtasks entry (Windows). Defaults daily at 03:00 with --from-wigle --chunk-size 10000, in dry-run on first install. Re-run --schedule without --schedule-dry-run to flip to live uploads. Marker comment in every artifact lets --unschedule clean up without touching anything else in your crontab / unit dir.
  • Bootstrap shim scripts: setup.sh / setup.bat / run.sh / run.bat / update.sh / update.bat. Double-click on Windows or run from a terminal — they handle the venv + deps + key save + timer install end-to-end.
  • 46 offline unit tests + scripts/smoke.sh. Tests block urllib.request.urlopen at the class level so any un-mocked live call fails loud. Smoke does a throwaway venv + deps + tests + a real systemd headless --schedule that asserts no --key/--wigle-key ever leaks into the rendered unit file.
  • CI matrix: Python 3.10 / 3.11 / 3.12.

Security

Scheduled commands never include --key / --wigle-key on the CLI — they read keys from the saved key files at run time. Unit files are world-readable via systemctl cat; baking a secret in there would leak it. Both the unit tests and the smoke script regression-net this.

Backward compatibility

All v1.1.x CLI flags, env vars, and on-disk paths are unchanged. Existing recipes from the README still run as written. The new flags are additive.

Live-tested

Platform Install Trigger Uninstall
Ubuntu 24.04 + systemd 255
Windows 11 Pro 24H2 + schtasks ✓ (239/261 chars under /TR limit)
Real WDGoWars key + synthetic CSV + --dry-run ✓ (multipart built, dry-run safety stop fired, hwm.json untouched) n/a n/a
CI matrix 3.10 / 3.11 / 3.12 n/a n/a

Upgrade

./update.sh                                          # bootstrap script — git pull + pip upgrade + version check
# or manually:
git pull && .venv/bin/pip install --upgrade -r requirements.txt

See CHANGELOG.md for the full release notes.

v1.1.2 — README install: PEP 668 / Bookworm fix

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 01 Jun 15:40
b1e35e2

README install snippets now create a project-local .venv/ instead of python3 -m pip install against the system Python.

Found by sweeping the feeder family after a Pi24 user reported the same crash in Muninn's setup.sh (Muninn v2.0.8, Heimdall v0.2.2). Wigle has no wrapper scripts to fix, so this is a docs-only release.

Fixed

  • README "Option A — ZIP download" and "Option B — clone with git" now use python3 -m venv .venv followed by .venv/bin/pip install and .venv/bin/python wigle_to_wdgwars.py. No more error: externally-managed-environment on Bookworm / Debian 12+ / Ubuntu 23.04+ / Homebrew Python.
  • "Updating" section matches.
  • Inline note explains what to do if python3 -m venv itself errors out (missing python3-venv apt package).

Unchanged

  • Windows install block — Windows Python has no PEP 668 enforcement.
  • wigle_to_wdgwars.py itself — no code change, only the version string.

Verified

Ran the new Option B block verbatim on Ubuntu 24.04 (same PEP 668 wall as Bookworm Pi24): venv create + pip install + --help with zero externally-managed errors.

Full changelog: https://github.com/HiroAlleyCat/wigle-to-wdgwars/blob/main/CHANGELOG.md

v1.1.0 — signed-JSON transport via gungnir

Choose a tag to compare

@HiroAlleyCat HiroAlleyCat released this 29 May 03:59

Structural refactor of the signed-JSON path onto gungnir 0.1.1 — the same library Muninn v2.0 sits on. Multipart CSV and WiGLE-side fetch stay local. Wire-protocol unchanged.

Free behavior wins on the signed-JSON path

  • Retry 5xx + network errors with exponential backoff (3 attempts).
  • 429 stops the batch and persists a cooldown the next cron tick respects.
  • Silent-drop detection (HTTP 200 ok:true with every counter zero now exits non-zero). v1.0 had no detection.
  • Inter-chunk cooldown of 1s (configurable via gungnir Client).
  • User-Agent now includes the repo URL per RFC bot-UA convention.
  • API-key redaction in log lines.
  • `--aircraft-batch` default dropped 1000 → 500 per locosp's 100-500 range.

Breaking changes

  • `whoami()` emits structured stderr logs (`key OK — user=X` + summary) instead of dumping raw JSON to stdout. Cron parsers that consumed the JSON must update.
  • `_post_signed()` return value: 200 on success, 1 on failure. The exact HTTP code for non-2xx is no longer surfaced (gungnir handles 5xx with retry internally).

Migration

```
pip install -r requirements.txt # new file, pulls gungnir from the pinned tag
```

API key at `~/.config/wigle-to-wdgwars/wdgwars.key` keeps working on every OS — local `load_key()` is preserved deliberately.

Verification

Live `--whoami` against wdgwars.pl: `key OK — user=AlleyCat` with correct account stats.