Measure the tracker on Ballroom, and close Q4

HANDOFF §8's last Phase 4 item. 698 clips, 6.1 hours, scored with
mir_eval at its 70 ms tolerance: beat F-measure 0.8941, downbeat
0.8575, in 95 seconds.

Ballroom is a set BeatNet+ was trained on — upstream's own README says
`| Ballroom | 699 | Train |`, and §A.2 agrees — so that number says the
port reproduces what the model can do over six hours of real audio, and
says nothing about generalisation. It cannot be held against §A.1's
80.62, which is GTZAN and GTZAN alone. evaluate.py now recognises the
training sets and prints that warning itself, so the mistake cannot be
made quietly.

Three findings worth the run on their own.

§5.3's default holds, and the margin is in the downbeats: generic
0.894/0.858, af-non-percussive 0.879/0.755, generic-main 0.793/0.581.

§5.5's octave fold is not free, and §5.5 does not say so. A fold cannot
change a beat F-measure at all — it relabels the tempo and the beats
still fire where the filter puts them, and the beat and downbeat scores
came out identical to four decimal places with the fold off, at 70-140
and at 88-176. Tempo is the only measure it can appear in, so --out now
carries the published tempo as a third column and the harness scores
MIREX's two tempo accuracies. There the fold cuts both ways: octave-
tolerant accuracy rises 0.936 -> 0.980, exact accuracy falls 0.769 ->
0.691, because Ballroom spans 87 to 204 BPM and no 2:1 window holds all
of it. Per genre it is stark: Samba 0.419 -> 0.953 and Waltz 0.509 ->
0.936 inside the window, Quickstep and Viennese Waltz 1.000 -> 0.000
outside it. The fold is a promise about the material, not a safety net,
and Phase 5's UI has to show that.

And the worst cases are not ours. On the hardest Samba files the tracker
calls 7 beats where the annotation has 51; the same activations through
upstream's own particle_filtering_cascade give 7, 7 and 8. It is
BeatNet+'s behaviour file for file — the filter only emits when the
median particle sits in the first four states of its beat, and a cloud
can agree on the tempo while spread across the phase. Upstream also
divides by a zero weight sum on that material and emits a NaN warning;
this port guards that case.

Q4, the DSP fallback: no. The neural path costs 86 us per 20 ms hop,
scores 0.89 on real audio and gives downbeat and meter, which no
classical ODF tracker does. The scaffold Q4 suggested building was never
built, so there is nothing to remove.

Author: Fohdeesha

README.md

@@ -50,6 +50,14 @@ detected meter as the quantum, timed through Link's own regression on the audio
 sample counter), a generic OSC namespace on any number of targets, and MIDI beat clock
 at 24 PPQN.
 
+Measured on the 698-clip Ballroom set — 6.1 hours, scored with mir_eval at its 70 ms
+tolerance — the whole chain gets **0.894 beat and 0.858 downbeat F-measure**. Ballroom is
+a set BeatNet+ was *trained* on, so that says the port reproduces what the model can do
+and nothing about how the model generalises; it is not comparable to BeatNet+'s published
+GTZAN figures. [tests/data/tracking/evaluation/](tests/data/tracking/evaluation/) has the
+numbers, the per-genre breakdown, what the octave fold costs as well as buys, and why the
+worst cases are BeatNet+'s behaviour rather than this port's.
+
 `takt4 --version` prints what it was built with, and the window opens; the UI does not
 show any of this yet.
 
@@ -140,18 +148,23 @@ python tools/pf_reference.py                                # regenerate the tra
 until this project's restatement of the particle filter has been held against the real
 one.
 
-`tools/evaluate.py` is the odd one out: it produces nothing committed. It measures beat
-and downbeat F-measure against an annotated dataset by running every file through
-`takt4-cli track` and scoring with mir_eval, so the number is the shipped C++ scored by
-the reference implementation the published figures come from. No dataset is vendored —
-they are large and their licences are their own — and the layouts of Ballroom, SMC and
-GTZAN work as they come:
+`tools/evaluate.py` measures beat and downbeat F-measure, and tempo accuracy, against an
+annotated dataset — by running every file through `takt4-cli track` and scoring with
+mir_eval, so the number is the shipped C++ scored by the reference implementation the
+published figures come from. No dataset is vendored — they are large and their licences
+are their own — and the layouts of Ballroom, SMC and GTZAN work as they come:
 
 ```sh
 pip install -r tools/requirements-eval.txt
 python tools/evaluate.py path/to/audio --annotations path/to/annotations --report eval.json
 ```
 
+It warns when the dataset it was pointed at is one BeatNet+ was trained on, because a
+score on those cannot be compared with the published ones. The trimmed reports of the
+runs behind the numbers above are committed in
+[tests/data/tracking/evaluation/](tests/data/tracking/evaluation/), which also says
+exactly where to get the Ballroom audio and annotations.
+
 Two steps because madmom builds from source and its `setup.py` imports numpy and
 Cython. Full-length source tracks belong outside git; `references/` is ignored for
 that purpose.

src/cli/main.cpp

@@ -113,8 +113,9 @@ void printUsage(std::ostream& out) {
         << "      --latency MS    added to every beat's timestamp; negative fires early\n"
         << "      --confidence T  hold the last tempo below this, default 0.15\n"
         << "      --seed N        the particle filter's seed, default 1\n"
-        << "      --out FILE      write every beat as <seconds> TAB <beat in bar>, the\n"
-        << "                      format the beat-tracking datasets annotate in\n"
+        << "      --out FILE      write every beat as <seconds> TAB <beat in bar> TAB\n"
+        << "                      <BPM>; the first two columns are the format the\n"
+        << "                      beat-tracking datasets annotate in\n"
         << "      --link          join the Ableton Link network as tempo master\n"
         << "      --osc H:P       send the generic namespace there; repeat for more\n"
         << "      --osc-prefix P  that namespace's prefix, default /takt4\n"
@@ -864,8 +865,12 @@ class Chain {
         }
     }
 
-    /// Writes every beat as "<seconds>\t<beat in bar>", the format the beat-tracking
-    /// datasets annotate in, so an estimate and a reference are the same kind of file.
+    /// Writes every beat as "<seconds>\t<beat in bar>\t<BPM>". The first two columns are
+    /// the format the beat-tracking datasets annotate in, so an estimate and a reference
+    /// are the same kind of file and the same reader handles both; the third is what the
+    /// tempo state machine was publishing at that beat, which is the only thing §5.5's
+    /// octave fold can move. A fold cannot change a beat time, so it cannot show up in
+    /// beat F-measure — the tempo column is where its effect is visible at all.
     void writeBeatsTo(const std::filesystem::path& path) {
         beatsOut_.open(path, std::ios::trunc);
         if (!beatsOut_) {
@@ -888,7 +893,7 @@ class Chain {
         }
         std::cout << beatLine(*event, tempo_.state()) << '\n';
         if (beatsOut_.is_open()) {
-            beatsOut_ << event->time << '\t' << event->beatInBar << '\n';
+            beatsOut_ << event->time << '\t' << event->beatInBar << '\t' << event->bpm << '\n';
         }
 
         if (osc_) {

tests/data/tracking/evaluation/README.md

@@ -0,0 +1,183 @@
+# Offline evaluation
+
+HANDOFF §8's last Phase 4 item: *"Offline evaluation against annotated audio; compare F1
+with §A.1's published numbers. Decide the fate of the DSP fallback (Q4)."*
+
+The five files here are `tools/evaluate.py`'s reports over the **Ballroom** set — 698
+clips, 6.1 hours — with the per-file scores stripped out; the configuration, the summary
+and the per-genre means are what a reader needs. Regenerate the full detail with
+`--report`. The audio and annotations are not vendored: see §"Reproducing" below.
+
+Every number is mir_eval's, at its default 70 ms tolerance, with its convention of
+trimming the first five seconds off both sides.
+
+## Read this before quoting any of it
+
+**Ballroom is a set BeatNet+ was trained on.** Upstream's own README lists it as
+`| Ballroom | 699 | Train |`, and HANDOFF §A.2 says the same. So the scores below say
+that this port reproduces what the model can do, over six hours of real audio — which is
+the strongest end-to-end check in the project, and the reason for running it — and they
+say **nothing** about how the model generalises.
+
+They must not be compared with §A.1's 80.62 beat / 56.51 downbeat. Those are GTZAN,
+which §A.2 records as *"test-only and never seen in training"*. A number that could be
+held against them needs GTZAN, and getting it is a separate decision (see the handoff).
+`tools/evaluate.py` prints this warning itself when it recognises the dataset, so the
+mistake cannot be made quietly.
+
+## What the tracker does
+
+`generic` weights, no octave fold — the configuration a published evaluation uses,
+because a fold needs a tempo range no evaluation may assume:
+
+| | mean | median |
+|---|---|---|
+| Beat F-measure | **0.8941** | 0.9223 |
+| Downbeat F-measure | **0.8575** | 0.9040 |
+| Tempo accuracy 1 | 0.7693 | 1.0000 |
+| Tempo accuracy 2 | 0.9355 | 1.0000 |
+| Cemgil | 0.8001 | 0.8274 |
+| Correct Metric Level Total | 0.7711 | 0.8196 |
+| P-score | 0.8615 | 0.9038 |
+
+Per genre, beat F-measure: Tango 0.961 and Viennese Waltz 0.961 at the top, **Samba
+0.796** at the bottom, with Rumba-Misc 0.836 next. 698 files in 95 seconds on the
+development machine, eight at a time.
+
+### The weight set: §5.3's default holds
+
+§5.3 makes `generic` the default and offers the other two by use case. On this material
+that is the right call by a wide margin, and the margin is in the downbeats:
+
+| weights | beat F1 | downbeat F1 |
+|---|---|---|
+| **generic** | **0.8941** | **0.8575** |
+| af-non-percussive | 0.8785 | 0.7546 |
+| generic-main | 0.7932 | 0.5805 |
+
+`af-non-percussive` is interesting rather than bad: its *median* beat F1 is 0.954,
+higher than `generic`'s 0.922, so it is better on most files and much worse on a few.
+`generic-main` — which upstream's README calls the main branch — is the weakest here on
+both measures, and its worst genre is Waltz at 0.574.
+
+### The octave fold is not free, and §5.5 does not say so
+
+§5.5: *"User sets an expected BPM range. Estimates outside it are ×2 / ÷2 folded into
+range before being considered. **This alone eliminates most real-world failures.**"*
+
+Two things are true and neither is obvious.
+
+**A fold cannot change a beat F-measure at all.** It relabels the tempo; the beats still
+fire wherever the particle filter puts them. Beat and downbeat F-measure came out
+*identical to four decimal places* with the fold off, at 70–140 and at 88–176. Tempo
+accuracy is the only measure a fold can appear in, which is why `--out` carries the
+published tempo as a third column and why the harness scores it.
+
+**And in tempo it cuts both ways:**
+
+| fold | Tempo accuracy 1 | Tempo accuracy 2 |
+|---|---|---|
+| off | **0.7693** | 0.9355 |
+| 88–176 | 0.6390 | 0.9628 |
+| 70–140 | 0.6905 | **0.9799** |
+
+Accuracy 2 forgives octave errors, and the fold helps it: fewer wild readings. Accuracy 1
+does not, and the fold *hurts* it — because Ballroom spans 87 to 204 BPM and no 2:1
+window contains all of that. Folding a correct answer out of range makes it wrong.
+
+Per genre, with the 70–140 fold, the mechanism is unmistakable:
+
+| genre | reference BPM | accuracy 1, no fold | accuracy 1, fold 70–140 |
+|---|---|---|---|
+| Rumba-Misc | 99 | 0.200 | **0.950** |
+| Samba | 101 | 0.419 | **0.953** |
+| Waltz | 87 | 0.509 | **0.936** |
+| Rumba-International | 100 | 0.627 | **1.000** |
+| ChaChaCha | 125 | 0.964 | 0.991 |
+| Rumba-American (7 files) | 128 | 1.000 | 0.857 |
+| Tango | 128 | 1.000 | 1.000 |
+| Jive | 171 | 0.967 | **0.100** |
+| Viennese Waltz | 178 | 1.000 | **0.000** |
+| Quickstep | 204 | 1.000 | **0.000** |
+
+Inside the window it is transformative — the slow genres are where the tracker doubles
+the tempo, and folding fixes almost all of it. Outside the window it is catastrophic:
+Quickstep and Viennese Waltz go from perfect to nothing.
+
+So the fold is a promise the operator makes about their material, not a safety net. For
+one DJ playing one genre it does what §5.5 claims. Across a set that spans more than an
+octave it destroys tempo readings that were already right. **Phase 5's UI has to make
+the window and its consequence visible** rather than leaving it as a slider with a
+plausible default — and it is worth considering whether the tracker should say when its
+unfolded estimate is far outside the window, because that is exactly when the fold is
+doing something drastic.
+
+### Where it fails, and whether that is ours
+
+Samba is the worst genre, and the worst individual files are Samba: on
+`Albums-Latin_Jam3-04` the tracker calls 7 beats where the annotation has 51. That looked
+like a port bug, so it was checked against upstream directly — the same activations
+through `particle_filtering_cascade`, three seeds:
+
+| file | annotated | takt4 | upstream |
+|---|---|---|---|
+| Albums-Latin_Jam3-04 | 51 | 7 | 7 / 7 / 8 |
+| Albums-Macumba-06 | 50 | 18 | 13 / 18 / 19 |
+| Albums-Macumba-08 | 50 | 11 | 10 / 7 / 12 |
+| Albums-Cafe_Paradiso-05 | 63 | 38 | 37 / 35 / 35 |
+| Media-103804 | 59 | 52 | 52 / 51 / 50 |
+
+It is BeatNet+'s behaviour, file for file. The cause is structural: the filter emits only
+when the *median* particle sits in the first four states of its beat, and a cloud that
+agrees on the tempo can still be spread across the phase — the tempo agreement on
+`Albums-Macumba-06` is 0.91 while nineteen seconds pass without a beat. Nothing in the
+port causes it and nothing in §5.4 asks for it to be fixed, but it is the failure an
+operator would notice, and it is the first thing to look at if beat *recall* ever becomes
+a goal.
+
+Upstream also emits `RuntimeWarning: invalid value encountered in divide` on this
+material — `weights / sum(weights)` with a zero sum. takt4 guards that case explicitly
+(`particle_filter.cpp`, "every hypothesis is impossible"), so the port is strictly more
+robust than the reference here.
+
+### Q4, the DSP fallback
+
+§3's Q4 asks whether to ship a classical tracker as a low-CPU alternative, and §8 puts
+the decision here. **No** — and the numbers make it easy. The neural path costs 86 µs of
+one core per 20 ms hop, scores 0.89 on real audio, and gives downbeat and meter, which no
+classical ODF tracker does. The recommendation in Q4 was to build BTrack as the Phase 1
+bring-up scaffold and then decide; the scaffold was never built (§6), so there is nothing
+to remove and no second tuning surface to support. If a machine is ever found that cannot
+afford 0.43% of a core, that is the time to revisit it.
+
+## Reproducing
+
+Neither the audio nor the annotations are vendored — they are large and their terms are
+their own. Both live under the git-excluded `references/`:
+
+```sh
+# annotations: 698 .beats files, "<time> TAB <beat in bar>", downbeat = 1
+git clone https://github.com/CPJKU/BallroomAnnotations references/ballroom-annotations
+
+# audio: 698 clips of 30 s, 1,453,888,083 bytes. Note https — the http:// URL in the
+# annotations README, and the page it points at, both return 403 now.
+curl -L -o references/ballroom-data1.tar.gz \
+  https://mtg.upf.edu/ismir2004/contest/tempoContest/data1.tar.gz
+mkdir -p references/ballroom-audio
+tar -xzf references/ballroom-data1.tar.gz -C references/ballroom-audio
+
+pip install -r tools/requirements-eval.txt
+python tools/evaluate.py references/ballroom-audio \
+    --annotations references/ballroom-annotations --jobs 8 \
+    --report references/ballroom.json
+```
+
+The annotations ask to be cited as Krebs, Böck & Widmer, ISMIR 2013. The audio is
+commercial ballroom-dance CDs distributed for the 2004 tempo contest with no licence
+attached and no redistribution right; decoding it locally to compute an F-measure is the
+ordinary research use, and `references/` is excluded by both `.gitignore` and
+`.git/info/exclude` so it cannot reach the repository.
+
+SMC MIREX was the other candidate and was not used: 217 deliberately-hard excerpts, but
+there is no public download — the MIREX pages use it as a private evaluation set and
+every trail ends at "contact the authors".

tests/data/tracking/evaluation/ballroom-af-non-percussive-nofold.json

@@ -0,0 +1,125 @@
+{
+  "annotations": "CPJKU/BallroomAnnotations @ 1db08914a8ae15edb01f104046e30bad88effe67",
+  "beatnet_plus_trained_on_this": [
+    "ballroom"
+  ],
+  "cli": "C:\\build\\takt4\\windows-core\\bin\\Release\\takt4-cli.exe",
+  "confidence_threshold": 0.15,
+  "dataset": "Ballroom (references/ballroom-audio), not vendored",
+  "failures": {},
+  "files": 698,
+  "octave_fold": "off",
+  "per_group": {
+    "ChaChaCha": {
+      "Downbeat F-measure": 0.7140272805229392,
+      "F-measure": 0.865484617617433,
+      "files": 111
+    },
+    "Jive": {
+      "Downbeat F-measure": 0.7070053064722093,
+      "F-measure": 0.9641259985587269,
+      "files": 60
+    },
+    "Quickstep": {
+      "Downbeat F-measure": 0.8785437033167688,
+      "F-measure": 0.9455639375724022,
+      "files": 82
+    },
+    "Rumba-American": {
+      "Downbeat F-measure": 0.904469420618489,
+      "F-measure": 0.9628032494166475,
+      "files": 7
+    },
+    "Rumba-International": {
+      "Downbeat F-measure": 0.777536290619455,
+      "F-measure": 0.8740771300010524,
+      "files": 51
+    },
+    "Rumba-Misc": {
+      "Downbeat F-measure": 0.6249991997758554,
+      "F-measure": 0.8055485400371593,
+      "files": 40
+    },
+    "Samba": {
+      "Downbeat F-measure": 0.6713311131343048,
+      "F-measure": 0.8614618691401944,
+      "files": 86
+    },
+    "Tango": {
+      "Downbeat F-measure": 0.8881759168687798,
+      "F-measure": 0.9443039662683405,
+      "files": 86
+    },
+    "VienneseWaltz": {
+      "Downbeat F-measure": 0.9482297364364638,
+      "F-measure": 0.9594863498421483,
+      "files": 65
+    },
+    "Waltz": {
+      "Downbeat F-measure": 0.602012420821619,
+      "F-measure": 0.732150688292278,
+      "files": 110
+    }
+  },
+  "platform": "Windows-10-10.0.19045-SP0",
+  "seed": 1,
+  "summary": {
+    "Any Metric Level Continuous": {
+      "files": 698,
+      "mean": 0.6951814521085492,
+      "median": 0.7763888888888889
+    },
+    "Any Metric Level Total": {
+      "files": 698,
+      "mean": 0.8241006099217918,
+      "median": 0.9056603773584906
+    },
+    "Cemgil": {
+      "files": 698,
+      "mean": 0.7787013110631436,
+      "median": 0.8474384972867772
+    },
+    "Cemgil Best Metric Level": {
+      "files": 698,
+      "mean": 0.8004541192278413,
+      "median": 0.8482039622121319
+    },
+    "Correct Metric Level Continuous": {
+      "files": 698,
+      "mean": 0.6701903681613056,
+      "median": 0.7604347826086957
+    },
+    "Correct Metric Level Total": {
+      "files": 698,
+      "mean": 0.786349848833477,
+      "median": 0.9047619047619048
+    },
+    "Downbeat F-measure": {
+      "files": 698,
+      "mean": 0.7545522044331278,
+      "median": 0.8571428571428572
+    },
+    "F-measure": {
+      "files": 698,
+      "mean": 0.8784973404362127,
+      "median": 0.9540229885057471
+    },
+    "Goto": {
+      "files": 698,
+      "mean": 0.505730659025788,
+      "median": 1.0
+    },
+    "Information gain": {
+      "files": 698,
+      "mean": 0.4792289410951436,
+      "median": 0.4733952287018425
+    },
+    "P-score": {
+      "files": 698,
+      "mean": 0.8588938011025233,
+      "median": 0.9272727272727272
+    }
+  },
+  "unmatched": [],
+  "weights": "af-non-percussive"
+}