Skip to content

Commit 505737e

Browse files
danjadanja
committed
many bits
1 parent bc923dd commit 505737e

11 files changed

Lines changed: 2365 additions & 41 deletions

‎docs/campione-profiles/00-overview.md‎

Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
# Sample Profiling System — Architecture Overview
2+
3+
**Status:** design, pre-implementation
4+
**Audience:** development agent team
5+
**Companion documents:** `01-descriptor-spec.md` (normative), `02-sample-profile-ontology.ttl` (normative)
6+
7+
---
8+
9+
## 1. Purpose
10+
11+
Given an audio sample — typically a single percussion hit, but the design does not
12+
assume this — produce a compact, deterministic, machine-readable *profile*: a fixed
13+
set of numeric descriptors serialised as RDF (Turtle).
14+
15+
Two consumers are in scope:
16+
17+
1. **Campione** (Downspout VST3 multi-zone sampler). When a set of drum samples is
18+
loaded, assign each to an appropriate General MIDI percussion note without user
19+
intervention. Constrained environment: portable C++, no heavyweight dependencies,
20+
runs at load time inside a plugin.
21+
2. **A standalone library tool** (later). Profile large sample collections, store the
22+
RDF in a triplestore, query and cluster it, and eventually train a learned
23+
classifier.
24+
25+
---
26+
27+
## 2. Core architectural decision: descriptor layer / decision layer
28+
29+
The two use cases differ only in *how a profile is turned into a label*, not in what a
30+
profile is. Therefore:
31+
32+
```
33+
┌──────────────────────────────────────────┐
34+
audio ────► │ DESCRIPTOR LAYER │ ────► profile
35+
│ deterministic, portable C++, no deps │ (RDF / vector)
36+
│ spec: 01-descriptor-spec.md │
37+
└──────────────────────────────────────────┘
38+
│
39+
┌───────────────────────────────────────────┤
40+
▼ ▼ ▼
41+
┌───────────────────┐ ┌────────────────────┐ ┌────────────────┐
42+
│ DECISION LAYER A │ │ DECISION LAYER B │ │ DECISION LAYER │
43+
│ rules + filename │ │ LLM via MCP │ │ C: trained │
44+
│ prior + Hungarian │ │ (oracle / dev aid) │ │ model │
45+
│ → Campione │ │ │ │ → standalone │
46+
└───────────────────┘ └────────────────────┘ └────────────────┘
47+
```
48+
49+
**The descriptor layer is written once and shared verbatim.** It is a static library
50+
with a C API and no dependency on the plugin framework, the RDF serialiser, or any
51+
classifier. The standalone tool links the same object code.
52+
53+
### Non-negotiable consequences
54+
55+
- **Versioned extractor.** Every profile records the extractor version and the analysis
56+
parameters used, via PROV-O. Descriptors computed by different versions are not
57+
comparable. Without this, the training corpus silently rots as the extractor evolves.
58+
- **Determinism.** Same input bytes + same extractor version ⇒ byte-identical Turtle
59+
output. No qualification: profiles use skolemised IRIs rather than blank nodes
60+
precisely so that this holds. This is a test requirement, not an aspiration.
61+
- **No classification logic in the descriptor layer.** No thresholds, no instrument
62+
names, no MIDI. If a classifier needs a new number, it is added to the spec and the
63+
version bumped.
64+
65+
---
66+
67+
## 3. Components
68+
69+
### 3.1 `libsampleprofile` — descriptor extraction
70+
71+
Portable C++17, header + single translation unit where practical. Dependencies: an FFT
72+
only (pffft or kissfft; Campione may already have suitable code behind its auto-pitch
73+
detection — check before adding one). No RDF dependency at this layer; serd belongs to
74+
§3.2 and the DSP core must build and test without it.
75+
76+
```c
77+
sp_profile_t sp_analyse(const float* const* channels,
78+
int channel_count,
79+
size_t frame_count,
80+
double sample_rate,
81+
const sp_params_t* params); /* NULL = defaults */
82+
```
83+
84+
Output is a plain struct of doubles — see §5 of the descriptor spec for the canonical
85+
65-element flat vector. Runtime target: under 20 ms for a 2-second 44.1 kHz stereo
86+
sample on modest hardware. Not real-time-safe; called from the loader thread.
87+
88+
### 3.2 `libsampleprofile-rdf` — Turtle reading and writing
89+
90+
Separate library so the DSP core stays dependency-free. Emits and consumes the shapes
91+
defined in `02-sample-profile-ontology.ttl`.
92+
93+
**Use [serd](https://drobilla.net/software/serd).** ISC-licensed, no dependencies
94+
beyond the C standard library, a few thousand lines, well under 100 KiB compiled,
95+
tested on Linux/macOS/Windows with GCC, Clang and MSVC, and explicitly designed for use
96+
inside plugins. Do not use Redland/librdf here: it drags in raptor2 and rasqal, which
97+
can want libxml2, libcurl and PCRE, and its LGPL option makes static linking into a
98+
VST3 binary a licensing conversation with no upside.
99+
100+
The decisive argument is the **reader**, not the writer. The `gm:alias` table (§3.3) is
101+
loadable Turtle data, so the plugin must parse Turtle — prefixed names, escapes,
102+
`@base` resolution, numeric literal forms. Hand-writing that is the worst option
103+
available. serd passes the full Turtle and TriG test suites and is fuzz-tested.
104+
105+
Scope it tightly:
106+
107+
- **serd only, no sord.** No in-memory model is needed. Stream the alias file through
108+
the reader callback into an `unordered_map`; stream profiles out through the writer.
109+
No SPARQL in the plugin, so rasqal is irrelevant.
110+
- **Number formatting is ours, not serd's.** The spec mandates 6-significant-figure
111+
output for cross-platform byte-identity. Format doubles in the extractor and pass
112+
them to serd as pre-serialised literal strings with explicit `^^xsd:double`. Do not
113+
hand serd a numeric value and accept whatever it prints.
114+
- **Skolemised IRIs, no blank nodes.** Profile, band-energy and envelope-segment nodes
115+
get real IRIs derived from the sample IRI — `<urn:sha256:3f7a1c…#profile>`,
116+
`#band-0`, `#env-3`. Output becomes byte-deterministic and diffable in git, Fuseki
117+
ingest stops having blank-node identity problems on reload, and SPARQL over the
118+
corpus gets substantially easier. Cost: nothing.
119+
120+
If content-addressed profiles are wanted later, RDFC-1.0 canonicalisation exists, but
121+
it is overkill for this shape of data.
122+
123+
On the standalone/corpus side, serd is not the natural choice — Fuseki is already in
124+
the picture and the ML work will be Python, so rdflib or Jena for query-heavy tooling,
125+
with the extractor's Turtle as the interchange format. Python bindings for serd exist
126+
if identical parsing on both sides matters more than convenience.
127+
128+
### 3.3 Filename tokeniser
129+
130+
Independent of the DSP. Takes a filename, returns zero or more scored
131+
`(gm:Instrument, confidence)` candidates.
132+
133+
In practice, filenames are the single strongest signal available: real-world packs are
134+
`Kick_01.wav`, `CH.wav`, `OHH_soft.wav`, `SD_rim.wav`. A tokeniser plus alias table
135+
gets a long way before any DSP runs. Treat the result as a **prior**, not an answer —
136+
acoustic evidence confirms or overrides it, and the RDF records which evidence drove
137+
the final call so that a wrong assignment is diagnosable.
138+
139+
Design notes:
140+
141+
- Split on `_`, `-`, space, camel-case boundaries, and digit runs.
142+
- Case-fold; strip index numbers and velocity-layer suffixes (`v1`, `vel3`, `_127`).
143+
- Alias table is **data, not code** — ship it as a Turtle file using `gm:alias`, so it
144+
can be extended without recompiling. Seed it from a survey of real sample packs.
145+
- Beware ambiguity: `rim` (37 side stick vs. 38 snare rim), `perc`, `hat` without
146+
open/closed qualifier, `tom` without pitch. Emit multiple candidates with split
147+
confidence rather than guessing.
148+
149+
### 3.4 Rule classifier (Campione)
150+
151+
Maps a profile to a scored candidate list over the GM percussion set. Rules operate on
152+
the descriptor vector; suggested first-cut discriminators, in rough order of value:
153+
154+
| Discriminator | Separates |
155+
|---|---|
156+
| `temporalCentroidNormalised`, `effectiveDuration` | short/decaying (kick, closed hat) vs. long ringing (crash, open hat, ride) |
157+
| `spectralFlatness.mean` | noise-like (snare, hats, cymbals) vs. pitched (kick, toms, cowbell, congas) |
158+
| `bandEnergy[0..1]` (20–120 Hz share) | kick vs. everything else |
159+
| `dominantPartial.frequency` + `salience` | kick (40–90 Hz) vs. floor/mid/high tom (80–250 Hz) |
160+
| `attackTailCentroidDelta` | pitch-swept membranes vs. static metal |
161+
| `bandEnergy[6..7]` (2.5 kHz+ share) | hats and cymbals vs. drums |
162+
| `envelopeSegment[*].rmsDb` shape | closed vs. open hat; gated vs. natural decay |
163+
| `decayFitR2` | single-exponential (damped drum) vs. multi-modal (cymbal) |
164+
165+
Do not hand-tune thresholds against intuition. Tune them against the labelled eval set
166+
(§4, step 5) and record the achieved accuracy per class.
167+
168+
### 3.5 Kit-level assignment
169+
170+
**Classify the kit, not the sample.** When *n* samples are loaded together,
171+
independent per-sample classification will cheerfully put three files on note 36. Build
172+
a cost matrix (samples × candidate GM slots, cost = negative log score) and solve the
173+
assignment with the Hungarian algorithm. *n* is small; cost is negligible. This single
174+
change removes the most visible failure mode.
175+
176+
Rules:
177+
- Allow a sample to remain unassigned if all costs exceed a rejection threshold.
178+
- Permit deliberate duplicates only where GM itself pairs them (35/36, 38/40, 49/57,
179+
51/59) and only when the scores are close.
180+
- Single-sample loads bypass the solver and take the argmax.
181+
182+
### 3.6 MCP integration
183+
184+
Campione already embeds an MCP HTTP server. Add two tools:
185+
186+
- `profile_zone(zone_index) → { turtle, vector }`
187+
- `classify_zone(zone_index, gm_note)` — apply/override an assignment
188+
189+
This gives, at near-zero cost, an LLM-as-classifier path: hand a client the descriptors
190+
plus the filename and let it choose the note. Useful in its own right, and more useful
191+
still as a cheap oracle for evaluating the rule classifier during development.
192+
193+
### 3.7 Corpus and learned classifier (later)
194+
195+
The deterministic extractor generates the corpus. Every time a user drags a zone to a
196+
different note, that correction is a ground-truth label — emit it as
197+
`sp:Classification` with `sp:evidence sp:UserAssignment` and push to Fuseki. After a few
198+
thousand samples there is a training set with a schema already attached, queryable by
199+
SPARQL for stratification and nearest-neighbour inspection.
200+
201+
The standalone application is then primarily a *graph* application — library-scale
202+
query, dedupe, clustering, similarity search — with the trained model as one more
203+
decision-layer plugin behind the same interface.
204+
205+
---
206+
207+
## 4. Build order
208+
209+
Each step is independently testable; do not start *n+1* before *n* passes.
210+
211+
1. **Freeze the descriptor spec.** Field names, units, FFT parameters, edge cases.
212+
Everything downstream depends on it. Reviewed and merged before code is written.
213+
2. **`libsampleprofile`.** With a reference test corpus: synthetic signals (sine bursts,
214+
white noise bursts, exponentially decaying sines, silence, DC, clipped material) with
215+
analytically known descriptor values. Assert to 1e-9.
216+
3. **Ontology file + serd reader/writer.** Validate output with `serdi` and
217+
`riot --validate`, plus a SHACL shapes file. Round-trip through Fuseki. Test the
218+
reader against a deliberately awkward alias file: non-ASCII tokens, escapes,
219+
`@base`, comments.
220+
4. **Filename tokeniser + alias table**, with its own unit tests over a list of real
221+
pack filenames.
222+
5. **Hand-label 100–200 samples** spanning at least four differently-produced packs
223+
(acoustic, 808/909, lo-fi, orchestral percussion). This is the eval set. It is the
224+
step that gets skipped and then regretted; it gates everything after it.
225+
6. **Rule classifier + Hungarian assignment**, measured against the eval set. Report
226+
per-class precision/recall, not just overall accuracy — the overall figure is
227+
dominated by kick and snare.
228+
7. **MCP tools**, Fuseki ingest, then the learned model.
229+
230+
---
231+
232+
## 5. Interfaces between teams
233+
234+
| Boundary | Contract |
235+
|---|---|
236+
| DSP ↔ everything | `sp_profile_t` struct and the 65-element vector ordering in spec §5 |
237+
| Serialiser ↔ consumers | The ontology file; SHACL shapes derived from it; skolemised IRI scheme |
238+
| Tokeniser ↔ classifier | `(IRI, confidence)` candidate list |
239+
| Classifier ↔ Campione | GM note number per zone + confidence + evidence set |
240+
| Campione ↔ store | Turtle over HTTP to Fuseki, graph per sample-source directory |
241+
242+
---
243+
244+
## 6. Open questions for the team
245+
246+
1. **FFT library.** Does Campione's existing pitch detector expose a reusable FFT? If so,
247+
what size/precision constraints does it impose?
248+
2. **Existing Turtle code.** Campione already saves and loads patch files as Turtle.
249+
*What does that use?* If it is hand-rolled string emission, this project should not
250+
add a second, parallel mechanism — migrate the patch I/O onto serd at the same time
251+
and have one RDF path in the codebase. Check this before anyone writes serialiser
252+
code; it may change the size and shape of the task considerably.
253+
3. **Sample identity.** Profiles need stable IRIs. Content hash (SHA-256 of the audio
254+
data, excluding metadata chunks) is proposed — it survives renaming and moving, which
255+
file IRIs do not. Confirm the cost is acceptable at load time.
256+
4. **Non-percussive input.** The descriptor set is percussion-oriented but not
257+
percussion-only. Should the profiler flag "probably not a one-shot" (long duration,
258+
sustained envelope, multiple onsets) rather than emit misleading temporal features?
259+
Recommendation: yes — a boolean `sp:multipleOnsets` and `sp:sustained`, cheap to
260+
compute, prevents downstream nonsense.
261+
5. **Namespace hosting.** `http://purl.org/stuff/sampleprofile/` and
262+
`http://purl.org/stuff/gm/` need PURL registration and content negotiation before
263+
first public release.
264+
6. **Alignment verification.** The ontology asserts a small number of mappings to the
265+
Audio Features Ontology, Music Ontology and EBUCore. These are marked in the file and
266+
must be checked against the live vocabularies before release; do not assume the term
267+
names are correct as written.

0 commit comments

Comments
 (0)