diff --git a/proposals/2026-08-03_rekordbox-usb-export.md b/proposals/2026-08-03_rekordbox-usb-export.md new file mode 100644 index 0000000..5f895fe --- /dev/null +++ b/proposals/2026-08-03_rekordbox-usb-export.md @@ -0,0 +1,272 @@ +# rekordbox USB Export (PDB/ANLZ) for CDJ/XDJ hardware + +* **Owners:** @sparkly-quasar (openboxxx) +* **Implementation Status:** Partially implemented. Standalone writer proven and verified against + independent parsers; in-Mixxx exporter working in a fork that compiles + links on macOS + (Intel + Apple Silicon), Windows, and Linux via CI. Pending rekordbox-desktop import + (verification tier 3) and physical CDJ/XDJ playback (tier 4). +* **Related Issues and PRs:** + * mixxxdj/mixxx#9463 — original "export crates/cues/beatgrids to rekordbox USB" request + * mixxxdj/mixxx#10321 — "Rekordbox USB Export (PDB/ANLZ)" + * mixxxdj/mixxx#12126 — "Abstract over library exporters" (the architectural hook) + * mixxxdj/mixxx#15556 — "Add OneLibrary support" (explicitly *out of scope* here; see Non-Goals) + * mixxxdj/mixxx#13293 — updated rekordbox Kaitai definitions (import-side) +* **Other docs or links:** + * Mixxx fork with the working exporter: + [sparkly-quasar/mixxx `feat/rekordbox-usb-export`](https://github.com/sparkly-quasar/mixxx/tree/feat/rekordbox-usb-export) + * Downloadable tri-platform CI builds: + [openboxxx Releases](https://github.com/sparkly-quasar/openboxxx/releases) + * openboxxx design docs + component status table: `libopenboxxx-export/README.md`; plus + architecture, export-design, legal, research-findings (linked inline) + * pyrekordbox (Dylan Jones, MIT) — ANLZ/PDB reference implementation + * Deep Symmetry `crate-digger` / `dysentery` `.ksy` specs (format documentation) + * rekordcrate (Jan Holthuis, MPL-2.0) — Rust PDB parser/serializer + +--- + +> **A note from the author.** I'm a long-time Mixxx user, not a professional C/C++ developer. This +> exporter grew out of experimenting with AI-assisted ("vibecoding") development, and getting a real +> stick to round-trip through the parsers felt like a genuinely worthwhile use of that time — so I'd +> like to offer it back to the community as a gift. I have no attachment to *my* code being the +> answer; if the right outcome is that this informs a rekordcrate-based path (or anything else the +> maintainers prefer), that's a win. I'm mainly hoping to move #9463 forward and to help however is +> most useful, including finding CDJ hardware testers. + +## TL;DR + +Mixxx can *import* rekordbox libraries but cannot *export* to the USB format that +Pioneer/AlphaTheta CDJ and XDJ hardware reads. This proposal adds a **rekordbox USB +exporter**: a writer for the unencrypted `export.pdb` (DeviceSQL) and per-track `ANLZ` +(`.DAT`/`.EXT`) analysis files, plus the `PIONEER/` USB directory layout and audio-file +copy — the same kind of "export my whole library to foreign DJ hardware" flow Mixxx +already ships for Engine DJ / Engine Prime. + +The core byte-writer is built as a **standalone, Qt-free C++ library** (`libopenboxxx-export`) +so the hard, fiddly serialization can be tested in isolation, with Mixxx integration as a thin +adapter modeled on the existing `EnginePrimeExportJob`. A **round-trip verifier** (parse our +own output back with Mixxx's in-tree Kaitai parsers and diff it) is a first-class feature, so a +DJ learns whether a stick is good *before* they leave for the gig. + +Scope is deliberately narrowed to the **legally clean, widest-reach** target: the classic +unencrypted PDB/ANLZ format. The newer encrypted OneLibrary / Device Library Plus format +(#15556) is explicitly a non-goal here for legal reasons (see Non-Goals). + +**This is not a paper design.** The standalone writer emits a full `export.pdb` (all 20 standard +tables + the 8-colour palette) and per-track ANLZ, every output cross-checked against two +independent real-format parsers and diffed field-for-field against genuine rekordbox files. An +in-Mixxx "Export Library to rekordbox USB" button already exists in a fork that compiles and links +on macOS (Intel + Apple Silicon), Windows, and Linux via CI — downloadable builds are on the +[Releases](https://github.com/sparkly-quasar/openboxxx/releases) page. What remains is the real +proof: importing into rekordbox desktop and playing on physical CDJ/XDJ hardware — testers welcome. + +## Why + +**Motivation and context.** "Export to rekordbox USB" has been an open, confirmed wishlist item +since 2018 (#9463), re-filed more specifically in 2021 (#10321), with no assignee, no branch, and +no merged PR. It is one of the most-requested missing features because it removes the last reason a +Mixxx user is forced back into rekordbox: preparing a set in open software but being unable to play +it on the club's CDJs. Mixxx already reads rekordbox libraries; the export direction is the missing +half. + +Much of the groundwork already exists in-tree, which makes the gap bounded rather than green-field: + +* A **working Engine DJ / Engine Prime exporter** (`Library ▸ Export Library to Engine Prime`) + establishes the exact pattern for walking the Mixxx library and writing a foreign DJ database. +* An **exporter-abstraction plan (#12126)** already intends to replace the Engine-specific menu item + with a generic `Library ▸ Export Library…` and a common per-format interface — the natural socket + for a rekordbox exporter. +* **rekordbox format definitions are already in-tree**: the importer uses Kaitai-generated + `rekordbox_pdb` / `rekordbox_anlz` parsers (`lib/rekordbox-metadata/`, kept current by #13293), so + the *read* side of PDB/ANLZ is already described. +* A **mature importer (since v2.3)** already works out the Mixxx↔rekordbox field mapping (hotcues, + memory cues, loops, colors, keys) in reverse; the exporter reuses that mapping the other direction. + +**Pitfalls of the current solution.** + +* **There is no export path at all** — a Mixxx user with a CDJ has to redo their prep in rekordbox. +* **Kaitai Struct can only READ.** Its C++ target generates parsers, not serializers, so Mixxx can + parse PDB/ANLZ today but has no generated code to *write* them. The single biggest chunk of + net-new code — a hand-written PDB/ANLZ serializer — simply does not exist yet in the project. +* **No abstraction to plug into yet.** The exporter interface envisioned in #12126 hasn't been built, + so today a new exporter would have to bolt onto the Engine-specific plumbing. + +## Goals + +* Add a **rekordbox USB exporter** that produces a stick a real CDJ/XDJ can browse and play from: + library, playlists, per-track beatgrids, and hot/memory cues. +* Deliver the serializer as a **standalone, dependency-light C++ library** with its own tests, so the + risky byte-layout work is verifiable independently of a Mixxx build and the eventual Mixxx PR is + mostly adapter + dialog + CMake option. +* Ship a **round-trip verifier** as a first-class trust feature: parse our own USB back with the + in-tree Kaitai parsers and report fidelity ("playlists 12/12, hotcues OK, beatgrids OK") before the + DJ relies on it. +* Land as an **upstream contribution**, implementing (and helping shape) the generic exporter + interface of #12126 rather than a one-off menu item — and, ideally, retiring #9463 / #10321. +* Provide an **in-app "report an export bug" flow** that opens a prefilled GitHub issue from a + structural diagnostic manifest (counts, sections written, verifier results, hashes — never the + user's audio or full metadata), so every beta tester becomes a hardware-compatibility data source. + +**Audience.** Mixxx users who play on Pioneer/AlphaTheta CDJ/XDJ hardware and currently keep +rekordbox around solely to export USB sticks; Mixxx maintainers who own the library-export subsystem; +and beta testers with access to real CDJ/XDJ units. + +## Non-Goals + +* **OneLibrary / Device Library Plus (`exportLibrary.db`, #15556) is out of scope — for legal, not + just effort, reasons.** That database is SQLCipher-encrypted (256-bit AES); writing it means + circumventing an access-control measure with an extracted key, which is precisely the DMCA §1201 + exposure the project is structured to avoid. It should be pursued **only** via an official + AlphaTheta spec/partnership, never with a recovered key. This proposal writes **only** the + unencrypted classic PDB/ANLZ format. (See openboxxx `docs/legal.md`.) +* **Reading the encrypted rekordbox 6/7 app database (`master.db`).** The sanctioned import path is + the user's own `rekordbox.xml`; we never decrypt their app DB. +* **Full analysis fidelity in the first milestone.** Waveforms (`PWAV`/`PWV3`/`PWV4`/`PWV5`), extended + `.EXT` cues with colors + labels (`PCO2`), `.2EX`, `PVBR`, My-Tags (`exportExt.pdb`), and history + playlists are deferred to a follow-up phase (they are additive ANLZ sections and don't touch the + PDB). The first milestone is a *playable* stick with a blank on-screen waveform, not zero playback. +* **Bundling or shipping any AlphaTheta-extracted key or crate-digger (EPL) code.** Format knowledge + is taken from public specs and permissively-licensed reference implementations only (see How ▸ + Licensing). + +## How + +**Overview.** A standalone C++ library takes a plain intermediate model and emits bytes; Mixxx +integration is a thin adapter that fills that model from the library. + +```text +Mixxx library (tracks + cues + beatgrids + crates/playlists [+ waveforms, phase 2]) + │ + ▼ ExportModel (plain structs — no Qt) + ┌──────────────────────────────────────────────────────────────────────┐ + │ libopenboxxx-export (standalone, Qt-free) │ + │ mapping/ Mixxx-model → rekordbox-model (colors, keys, ms↔frame,…) │ + │ pdb/ export.pdb writer — LITTLE-endian DeviceSQL pages │ + │ anlz/ ANLZ .DAT/.EXT — BIG-endian tagged sections │ + │ usb/ PIONEER/ dir layout, ANLZ path assignment, audio copy │ + │ verify/ round-trip: parse our output w/ in-tree Kaitai + diff │ + │ diag/ structural diagnostic bundle (feeds in-app bug report) │ + └──────────────────────────────────────────────────────────────────────┘ + │ + ▼ + USB stick: /PIONEER/… + copied audio files +``` + +The Mixxx-side adapter is a `RekordboxExportJob` that mirrors the existing `EnginePrimeExportJob`: +a `QThread` with `loadIds/loadTrack/loadCrate/loadPlaylist` marshalling and +`jobMaximum/jobProgress/completed/failed` signals, driving an export dialog, all behind a +`REKORDBOX_EXPORT` CMake option. This is the concrete implementation of the exporter abstraction +discussed in #12126, so the proposal both uses and helps define that interface. (The fork also adds +a small "Cue Sheet to Tracklist" utility alongside it.) The fork compiles and links on macOS +(Intel + Apple Silicon), Windows, and Linux under CI today. + +**Cross-platform by construction.** Byte layout is always explicit, never host-dependent: PDB integers +written explicitly little-endian, ANLZ integers explicitly big-endian (separate helpers, never +shared); strings encoded by hand per `device_sql_string` rules (short-ASCII mangling or `0x90` +UTF-16**LE** for PDB, UTF-16**BE** for ANLZ paths/labels — no `std::wstring`/`wchar_t`); +USB-internal paths always forward-slashed regardless of host OS; no FAT/exFAT-specific code (the DJ +formats the stick). This is why macOS/Linux/Windows are all supported from day one, which also +maximizes the beta-tester (hardware data) pool. + +**Testing and verification.** A four-tier ladder, cheapest first: + +1. **Round-trip self-check** — write a stick, parse it back with the in-tree Kaitai parsers, confirm + it reproduces the intended model. Free; runs on every build in CI. +2. **Diff vs a real rekordbox stick** — structural field comparison against a genuine + rekordbox-produced stick, to catch layout mistakes the parser tolerates. +3. **Import into rekordbox desktop** — behavioral smoke test. ⚠️ *Not* equivalent to hardware (the + desktop app is more forgiving and may re-analyze). Standardize on **rekordbox 5** for this tier: + a rekordbox 7 stick writes `export.pdb` as an empty compatibility shell and puts the real library + in the encrypted OneLibrary DB, so rb7 is a misleading oracle for classic PDB. +4. **Real CDJ/XDJ hardware** — the only true proof, run at milestones. Launch matrix: CDJ-2000NXS2 + (legacy baseline), CDJ-3000/3000X (strict/modern), XDJ-AZ (modern all-in-one). + +Tiers 1–2 are exposed as a **library API** (`verify/`) and reused by the in-app verifier so the DJ +gets the same check. **Current status: tiers 1–2 are green** — exports complete without errors and +round-trip through both independent parsers. **Tiers 3–4 (rekordbox-desktop import and physical +CDJ/XDJ playback) are the outstanding proof and have not yet been run**; this is the main thing +hardware testers can help unblock. + +**Migration and downtime.** None — this is purely additive. It introduces a new +export path (generalizing the existing Engine Prime menu item) behind a `REKORDBOX_EXPORT` CMake +option, in the same spirit as the existing `ENGINEPRIME`/`__ENGINEPRIME__` gate. No existing data, +schema, or user workflow changes; +nothing is removed. Existing rekordbox *import* is untouched. + +**Licensing.** The module is **GPLv2** (Mixxx-compatible). ANLZ layout is *ported* from **pyrekordbox** +(MIT → GPLv2-compatible; attributed). PDB layout is implemented from the **crate-digger `.ksy` spec** +(EPL-1.0) used as a *format description*, not copied code — the same clean-room-ish posture by which +Mixxx's existing rekordbox importer was built (a file format is not itself copyrightable; a specific +code expression is). crate-digger's EPL code is never linked. rekordcrate (MPL-2.0) is +GPL-compatible and is discussed as an alternative below. + +**Known unknowns / open questions.** + +* `track_row` "always X" magic constants and `bitmask` semantics — written from reference values; + worth confirming behaviorally on hardware. +* All 20 standard PDB tables (including the library-independent `columns` / `unknown_17` / + `unknown_18` / `history` browse/sort menu tables) are now written, and output round-trips through + both parsers. The remaining unknown is purely behavioral: whether CDJ firmware is satisfied with the + exact contents — resolved only at tiers 3–4. +* The **ANLZ-path hash quirk** (flagged by a prior community exporter): confirm whether newer firmware + requires a specific folder-path hash before assuming free choice of `USBANLZ` folder names. +* PDB page conventions and `first_page`/`last_page` semantics — pass the parsers; validate on hardware. + +## Alternatives + +1. **Use rekordcrate (Rust) for PDB export instead of a C++ writer.** rekordcrate (Jan Holthuis) is + a maintainer-suggested path and already models the PDB format in Rust with serialization in view. + *Objection to this proposal:* "why hand-write a C++ serializer when a Rust one exists?" + *Counter-arguments:* (a) Mixxx is C++/CMake with no Rust in the build today, so adopting + rekordcrate means introducing a Rust toolchain + FFI boundary into the core build — a larger + architectural commitment than the exporter itself; (b) rekordcrate covers PDB but not the full + ANLZ writer path, USB layout, verifier, and diagnostic bundle this proposal needs, so it is a + partial solution regardless; (c) a Qt-free C++ library keeps the serializer inside the language + and build system maintainers already use. **This is genuinely an open decision for maintainers**, + and the standalone-library structure here is deliberately arranged so the writer could be swapped + for a rekordcrate-backed one without disturbing the Mixxx adapter — the adapter/dialog/CMake work + is reusable either way. Maintainer direction on C++-writer vs Rust-via-rekordcrate is explicitly + solicited. +2. **Generate a serializer from Kaitai instead of hand-writing.** Rejected: Kaitai's C++ write + support is experimental/partial and not viable today; the read parsers stay Kaitai-generated, the + writers are hand-written. +3. **Hard-fork Mixxx and add export out-of-tree.** Rejected as the primary path: forking splits the + community and forfeits Mixxx's test base, review, and credibility. Upstreaming into the existing + export subsystem is preferred; a fork is a fallback only if upstream cannot accept the work. +4. **Export `rekordbox.xml` only, and let users import that into rekordbox.** Insufficient: XML round- + trips metadata but does not produce a USB a CDJ can play from standalone — it still requires the + user to open rekordbox and re-export to USB, which is the exact dependency this feature removes. +5. **Target OneLibrary because it's the "newest, sanctioned" format.** Rejected on legal grounds — it + is encrypted; see Non-Goals. Counterintuitively the newest format is the one we must *not* + reverse-engineer key-in-hand. + +## Action Plan + +* [x] Survey the state of the Mixxx export effort and confirm the in-tree hooks (Engine DJ exporter, + `lib/rekordbox-metadata/`, #12126). — openboxxx `docs/mixxx-export-status.md`, + `research-findings.md` +* [x] **Phase 0 — standalone PDB/ANLZ writer + verifier, proven.** `libopenboxxx-export` emits a full + `export.pdb` (all 20 standard tables + 8-colour palette) and per-track ANLZ (beat grids + hot/ + memory cues); every output is cross-checked against two independent parsers (pyrekordbox for + ANLZ, a crate-digger-spec Kaitai parser for PDB) and diffed field-for-field against genuine + rekordbox ANLZ files. +* [x] **Phase 1 reader path.** `openboxxx_from_mixxx` reads a real `mixxxdb.sqlite` into the export + model with no Mixxx build required — validated on a ~2,900-track library. +* [x] **Phase 1 in-Mixxx exporter, working in a fork.** "Export Library to rekordbox USB" button — + `RekordboxExportJob` + dialog mirroring the Engine DJ exporter, behind a `REKORDBOX_EXPORT` + CMake option (plus a "Cue Sheet to Tracklist" tool). Compiles + links on macOS (Intel + Apple + Silicon), Windows, and Linux via CI; downloadable builds on the + [Releases](https://github.com/sparkly-quasar/openboxxx/releases) page. — refs mixxxdj/mixxx#12126 +* [ ] **Verification tier 3 — rekordbox desktop import.** Confirm a generated stick imports cleanly + into rekordbox 5 (the classic-PDB target; not rb7, which prefers the encrypted OneLibrary DB). +* [ ] **Verification tier 4 — real CDJ/XDJ hardware.** The true proof, on the launch matrix + (CDJ-2000NXS2, CDJ-3000/3000X, XDJ-AZ). Hardware testers very welcome. — refs mixxxdj/mixxx#9463, + mixxxdj/mixxx#10321 +* [ ] Resolve the residual behavioral unknowns at tiers 3–4 (ANLZ-path hash quirk, `track_row` magic + constants, exact firmware expectations for the menu tables). +* [ ] Propose/align the shared exporter interface upstream so this lands as a contribution, not a + fork. — refs mixxxdj/mixxx#12126 +* [ ] Add the in-app "report an export bug" flow (structural diagnostic bundle → prefilled GitHub + issue; no telemetry server; user reviews before it leaves the machine). +* [ ] Publish a hardware compatibility matrix with beta-tester results. +* [ ] Phase 2 (follow-up, additive): waveforms and extended `.EXT`/`PCO2` cues with colors + labels, + `.2EX` for CDJ-3000, `PVBR`.