Skip to content

Commit 7630a45

Browse files
dydy
committed
Add research & todo
1 parent 726bcef commit 7630a45

2 files changed

Lines changed: 555 additions & 0 deletions

File tree

‎.work/research.md‎

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,294 @@ IDEA: make openly available JS packages have classified analogs compiled with JZ
5353
| **Pro audio** | Ableton, Steinberg, PreSonus, Dolby, BBC R&D | Consulting contracts — "we build the open-source primitive, you fund the work" |
5454
| **Academic/research** | Stanford CCRMA, MIT Media Lab, IRCAM | Interactive demos as teaching tools; algorithm paper citations; research partnerships |
5555

56+
## Goal
57+
58+
prepare for **grants** (NLnet, Sovereign Tech Fund, Sloan, Google Web Fund) + **OSS corporate sponsors** (videojs/babel model) + **freemium hosted product** (cleanvoice/auphonic model). Three audiences, one site.
59+
60+
5661
## Refs
5762

5863
* https://github.com/webprofusion/OpenAudio
64+
* https://g-meh.com/
65+
66+
---
67+
68+
## `@audio` Package Ecosystem
69+
70+
### Design Principles
71+
72+
1. **audiojs = audio production & music DSP** — DAW/plugin language, not generic signal processing.
73+
2. **scijs = generic DSP primitives** — filters, transforms, windows, math.
74+
3. **audiojs depends on scijs** — wrappers and re-exports, not duplication.
75+
4. **umbrellas re-export atoms** — `@audio/pitch` → `@audio/pitch-yin`, `@audio/pitch-mcleod` etc.
76+
5. **atoms are installable standalone** — `@audio/pitch-yin` works without `@audio/pitch`.
77+
78+
### `@audio` Atoms & Umbrellas
79+
80+
#### `@audio/pitch` — Pitch Detection
81+
| Atom | Source |
82+
|------|--------|
83+
| `@audio/pitch-yin` | `pitch-detection/yin.js` |
84+
| `@audio/pitch-mcleod` | `pitch-detection/mcleod.js` |
85+
| `@audio/pitch-pyin` | `pitch-detection/pyin.js` |
86+
| `@audio/pitch-hps` | `pitch-detection/hps.js` |
87+
| `@audio/pitch-cepstrum` | `pitch-detection/cepstrum.js` |
88+
| `@audio/pitch-swipe` | `pitch-detection/swipe.js` |
89+
| `@audio/pitch-autocorrelation` | `pitch-detection/autocorrelation.js` |
90+
| `@audio/pitch-amdf` | `pitch-detection/amdf.js` |
91+
92+
#### `@audio/beat` — Beat Detection
93+
| Atom | Source |
94+
|------|--------|
95+
| `@audio/beat-detect` | `beat-detection/detect.js` |
96+
| `@audio/beat-onset` | `beat-detection/onset/` |
97+
| `@audio/beat-tempo` | `beat-detection/tempo/` |
98+
| `@audio/beat-track` | `beat-detection/track.js` |
99+
100+
#### `@audio/shift` — Pitch Shifting
101+
| Atom | Source |
102+
|------|--------|
103+
| `@audio/shift-ola` | `pitch-shift/ola.js` |
104+
| `@audio/shift-wsola` | `pitch-shift/wsola.js` |
105+
| `@audio/shift-pvoc` | `pitch-shift/phase-lock.js` |
106+
| `@audio/shift-pvlock` | `pitch-shift/phase-lock.js` (phase-locked variant) |
107+
| `@audio/shift-transient` | `pitch-shift/transient.js` |
108+
| `@audio/shift-psola` | `pitch-shift/psola.js` |
109+
| `@audio/shift-granular` | `pitch-shift/granular.js` |
110+
| `@audio/shift-paulstretch` | `pitch-shift/paulstretch.js` |
111+
| `@audio/shift-sms` | `pitch-shift/sms.js` |
112+
| `@audio/shift-hpss` | `pitch-shift/hpss.js` |
113+
| `@audio/shift-formant` | `pitch-shift/formant-shift.js` |
114+
| `@audio/shift-sample` | `pitch-shift/sample.js` |
115+
| `@audio/shift-hybrid` | `pitch-shift/hybrid.js` |
116+
117+
#### `@audio/stretch` — Time Stretching
118+
| Atom | Source |
119+
|------|--------|
120+
| `@audio/stretch-wsola` | `time-stretch/wsola.js` |
121+
| `@audio/stretch-pvoc` | `time-stretch/vocoder.js` |
122+
| `@audio/stretch-pvlock` | `time-stretch/` (phase-locked variant) |
123+
| `@audio/stretch-transient` | `time-stretch/` (transient-aware variant) |
124+
| `@audio/stretch-paulstretch` | `time-stretch/paulstretch.js` |
125+
| `@audio/stretch-psola` | `time-stretch/psola.js` |
126+
| `@audio/stretch-sms` | `time-stretch/sms.js` |
127+
128+
#### `@audio/filter` — Audio Filters (DAW/plugin language)
129+
Re-exports curated `digital-filter` primitives with audio-friendly parameter names (Hz, Q, dB, type).
130+
131+
| Export | Source | Audio Concept |
132+
|--------|--------|---------------|
133+
| `highpass` | `digital-filter/iir/biquad.js` | High-pass filter |
134+
| `lowpass` | `digital-filter/iir/biquad.js` | Low-pass filter |
135+
| `bandpass` | `digital-filter/iir/biquad.js` | Band-pass filter |
136+
| `notch` | `digital-filter/iir/biquad.js` | Notch filter |
137+
| `allpass` | `digital-filter/iir/biquad.js` | All-pass filter |
138+
| `resonator` | `audio-filter/effect/resonator.js` | Resonant filter |
139+
| `comb` | `audio-filter/effect/comb.js` | Comb filter |
140+
| `dcBlocker` | `audio-filter/effect/dc-blocker.js` | DC offset removal |
141+
| `variableBandwidth` | `audio-filter/effect/variable-bandwidth.js` | Variable bandwidth filter |
142+
143+
**Note**: Generic filter design (Butterworth, Chebyshev, FIR design, etc.) stays in `digital-filter` (scijs).
144+
145+
#### `@audio/eq` — Equalization & Tone Shaping
146+
| Atom | Source |
147+
|------|--------|
148+
| `@audio/eq-highshelf` | `audio-filter/eq/highshelf.js` |
149+
| `@audio/eq-lowshelf` | `audio-filter/eq/lowshelf.js` |
150+
| `@audio/eq-parametric` | `audio-filter/eq/parametric-eq.js` |
151+
| `@audio/eq-graphic` | `audio-filter/eq/graphic-eq.js` |
152+
| `@audio/eq-baxandall` | `audio-filter/eq/baxandall.js` |
153+
| `@audio/eq-tilt` | `audio-filter/eq/tilt.js` |
154+
| `@audio/eq-crossover` | `audio-filter/eq/crossover.js` |
155+
156+
#### `@audio/weighting` — Frequency Weighting Curves
157+
| Atom | Source |
158+
|------|--------|
159+
| `@audio/weighting-a` | `audio-filter/weighting/a-weighting.js` |
160+
| `@audio/weighting-c` | `audio-filter/weighting/c-weighting.js` |
161+
| `@audio/weighting-k` | `audio-filter/weighting/k-weighting.js` |
162+
| `@audio/weighting-itu468` | `audio-filter/weighting/itu468.js` |
163+
| `@audio/weighting-riaa` | `audio-filter/weighting/riaa.js` |
164+
165+
#### `@audio/auditory` — Auditory Filterbanks
166+
| Atom | Source |
167+
|------|--------|
168+
| `@audio/auditory-bark` | `audio-filter/auditory/bark-bank.js` |
169+
| `@audio/auditory-erb` | `audio-filter/auditory/erb-bank.js` |
170+
| `@audio/auditory-mel` | `audio-filter/auditory/mel-bank.js` |
171+
| `@audio/auditory-gammatone` | `audio-filter/auditory/gammatone.js` |
172+
173+
#### `@audio/effect` — Audio Effects
174+
| Atom | Source |
175+
|------|--------|
176+
| `@audio/effect-phaser` | `audio-effect/modulation/phaser.js` |
177+
| `@audio/effect-flanger` | `audio-effect/modulation/flanger.js` |
178+
| `@audio/effect-chorus` | `audio-effect/modulation/chorus.js` |
179+
| `@audio/effect-wah` | `audio-effect/modulation/wah.js` |
180+
| `@audio/effect-autowah` | `audio-effect/modulation/auto-wah.js` |
181+
| `@audio/effect-tremolo` | `audio-effect/modulation/tremolo.js` |
182+
| `@audio/effect-vibrato` | `audio-effect/modulation/vibrato.js` |
183+
| `@audio/effect-ringmod` | `audio-effect/modulation/ring-mod.js` |
184+
| `@audio/effect-freqshift` | `audio-effect/modulation/frequency-shifter.js` |
185+
| `@audio/effect-delay` | `audio-effect/delay/delay.js` |
186+
| `@audio/effect-multitap` | `audio-effect/delay/multitap.js` |
187+
| `@audio/effect-pingpong` | `audio-effect/delay/ping-pong.js` |
188+
| `@audio/effect-reverb` | `audio-effect/delay/reverb.js` |
189+
| `@audio/effect-distortion` | `audio-effect/distortion/distortion.js` |
190+
| `@audio/effect-bitcrusher` | `audio-effect/distortion/bitcrusher.js` |
191+
| `@audio/effect-exciter` | `audio-effect/distortion/exciter.js` |
192+
| `@audio/effect-gain` | `audio-effect/utility/gain.js` |
193+
| `@audio/effect-mixer` | `audio-effect/utility/mixer.js` |
194+
| `@audio/effect-slew` | `audio-effect/utility/slew-limiter.js` |
195+
| `@audio/effect-nshaper` | `audio-effect/utility/noise-shaping.js` |
196+
197+
#### `@audio/speech` — Speech Processing
198+
| Atom | Source |
199+
|------|--------|
200+
| `@audio/speech-formant` | `audio-filter/speech/formant.js` |
201+
| `@audio/speech-lpc` | `audio-filter/speech/lpc.js` |
202+
| `@audio/speech-vocoder` | `audio-filter/speech/vocoder.js` |
203+
204+
#### `@audio/denoise` — Noise Reduction & Restoration
205+
| Atom | Source |
206+
|------|--------|
207+
| `@audio/denoise-gate` | `noise-reduction/gate.js` |
208+
| `@audio/denoise-spectral` | `noise-reduction/specsub.js` |
209+
| `@audio/denoise-wiener` | `noise-reduction/wiener.js` |
210+
| `@audio/denoise-omlsa` | `noise-reduction/omlsa.js` |
211+
| `@audio/denoise-dehum` | `noise-reduction/dehum.js` |
212+
| `@audio/denoise-declick` | `noise-reduction/declick.js` |
213+
| `@audio/denoise-decrackle` | `noise-reduction/decrackle.js` |
214+
| `@audio/denoise-declip` | `noise-reduction/declip.js` |
215+
| `@audio/denoise-dewind` | `noise-reduction/dewind.js` |
216+
| `@audio/denoise-deplosive` | `noise-reduction/deplosive.js` |
217+
| `@audio/denoise-deesser` | `noise-reduction/deesser.js` |
218+
| `@audio/denoise-debreath` | `noise-reduction/debreath.js` |
219+
| `@audio/denoise-dereverb` | `noise-reduction/dereverb.js` |
220+
| `@audio/denoise-detect` | `noise-reduction/denoise.js` (noise-type classifier) |
221+
222+
#### `@audio/synth` — Synthesis
223+
| Atom | Source |
224+
|------|--------|
225+
| `@audio/synth-osc` | `periodic-function/*` — sine, sawtooth, square, triangle, trapezoid, pulse, clausen, noise, wavetable |
226+
| `@audio/synth-pluck` | Karplus-Strong plucked string synthesis |
227+
| `@audio/synth-rhythm` | Tempo-based rhythm generation with patterns |
228+
| `@audio/synth-chirp` | Linear/log frequency sweep generator |
229+
| `@audio/synth-dtmf` | Telephone dial tone generator |
230+
| `@audio/synth-risset` | Inharmonic drum synthesis |
231+
232+
#### Existing `@audio` Umbrellas
233+
| Package | Atoms |
234+
|---------|-------|
235+
| `@audio/speaker` | `@audio/speaker-darwin-arm64`, `@audio/speaker-darwin-x64`, `@audio/speaker-linux-arm64`, `@audio/speaker-linux-x64`, `@audio/speaker-win32-x64` |
236+
| `@audio/decode` | `@audio/decode-aac`, `@audio/decode-aiff`, `@audio/decode-amr`, `@audio/decode-caf`, `@audio/decode-flac`, `@audio/decode-mp3`, `@audio/decode-opus`, `@audio/decode-qoa`, `@audio/decode-vorbis`, `@audio/decode-wav`, `@audio/decode-webm`, `@audio/decode-wma` |
237+
| `@audio/encode` | `@audio/encode-aiff`, `@audio/encode-flac`, `@audio/encode-mp3`, `@audio/encode-ogg`, `@audio/encode-opus`, `@audio/encode-wav` |
238+
| `@audio/mic` | `@audio/mic-darwin-arm64`, `@audio/mic-darwin-x64`, `@audio/mic-linux-arm64`, `@audio/mic-linux-x64`, `@audio/mic-win32-x64` |
239+
240+
#### `@audio/dynamics` — Dynamics Processing
241+
| Atom | Source |
242+
|------|--------|
243+
| `@audio/dynamics-compressor` | `audio-effect/dynamics/compressor.js` |
244+
| `@audio/dynamics-limiter` | `audio-effect/dynamics/limiter.js` |
245+
| `@audio/dynamics-expander` | `audio-effect/dynamics/expander.js` |
246+
| `@audio/dynamics-gate` | `audio-effect/dynamics/gate.js` |
247+
| `@audio/dynamics-transient` | `audio-effect/dynamics/transient-shaper.js` |
248+
| `@audio/dynamics-envelope` | `audio-effect/dynamics/envelope.js` |
249+
250+
#### `@audio/vocals` — Vocal Isolation
251+
| Atom | Source |
252+
|------|--------|
253+
| `@audio/vocals-isolate` | `audio/fn/vocals.js` (center isolate) |
254+
| `@audio/vocals-remove` | `audio/fn/vocals.js` (center remove) |
255+
256+
#### `@audio/spatial` — Spatial / Channel Utilities
257+
| Atom | Source |
258+
|------|--------|
259+
| `@audio/spatial-midside` | Mid/Side encode/decode |
260+
| `@audio/spatial-channelsplit` | Channel split/combine |
261+
| `@audio/spatial-delay` | Per-channel delay |
262+
| `@audio/spatial-crossfeed` | Headphone crossfeed (`audio/fn/crossfeed.js`) |
263+
| `@audio/spatial-panner` | Panner (`audio-effect/spatial/panner.js`) |
264+
| `@audio/spatial-widen` | Stereo widener (`audio-effect/spatial/stereo-widener.js`) |
265+
| `@audio/spatial-haas` | Haas effect (`audio-effect/spatial/haas.js`) |
266+
267+
#### `@audio/spectral` — Spectral Features
268+
| Atom | Source |
269+
|------|--------|
270+
| `@audio/spectral-centroid` | Spectral centroid |
271+
| `@audio/spectral-flatness` | Spectral flatness |
272+
| `@audio/spectral-crest` | Spectral crest factor |
273+
| `@audio/spectral-rolloff` | Spectral rolloff |
274+
| `@audio/spectral-flux` | Spectral flux |
275+
| `@audio/spectral-slope` | Spectral slope |
276+
| `@audio/spectral-spread` | Spectral spread |
277+
278+
#### `@audio/mir` — Music Information Retrieval
279+
| Atom | Source |
280+
|------|--------|
281+
| `@audio/mir-structure` | Structural segmentation (verse/chorus/bridge) |
282+
| `@audio/mir-transcribe` | Polyphonic transcription (audio → MIDI) |
283+
| `@audio/mir-downbeat` | Downbeat estimation |
284+
| `@audio/mir-coversong` | Cover song identification |
285+
| `@audio/mir-melody` | Continuous melody F0 contour |
286+
| `@audio/mir-multif0` | Multiple simultaneous pitch estimation |
287+
| `@audio/mir-chroma` | `pitch-detection/chroma.js` |
288+
| `@audio/mir-chord` | `pitch-detection/chord.js` |
289+
| `@audio/mir-key` | `pitch-detection/key.js` |
290+
| `@audio/mir-genre` | Genre classification |
291+
| `@audio/mir-mood` | Mood/emotion classification |
292+
| `@audio/mir-tags` | Semantic audio tagging |
293+
| `@audio/mir-fingerprint` | Audio fingerprinting |
294+
| `@audio/mir-similarity` | Audio similarity metric |
295+
| `@audio/mir-drums` | Drum transcription |
296+
| `@audio/mir-lyrics` | Lyrics-to-audio alignment |
297+
| `@audio/mir-separate` | Stem separation (vocals/drums/bass/other) |
298+
| `@audio/mir-tonnetz` | Tonal centroid features |
299+
| `@audio/mir-tempogram` | Tempo over time |
300+
301+
### What Stays in `audio` Core
302+
303+
These are structural, page-based, or tightly coupled to the `audio` engine. Not atomized.
304+
305+
**Structural ops:** crop, trim, split, insert, remove, repeat, pad, reverse, mix, remix, loop, crossfade, speed
306+
**Basic stats:** peak, rms, loudness (LUFS), silence, clipping, DC offset, duration, sampleRate, channels
307+
**Utilities:** gain, fade, normalize, pan, write, save, encode, decode, play, stream, record
308+
309+
### Naming Policy
310+
311+
| Namespace | Meaning |
312+
|-----------|---------|
313+
| `@audio/pitch-*` | Pitch detection algorithms (F0 estimation) |
314+
| `@audio/beat-*` | Beat/onset/tempo detection |
315+
| `@audio/shift-*` | Pitch-shifting algorithms |
316+
| `@audio/stretch-*` | Time-stretching algorithms |
317+
| `@audio/filter` | Audio-facing filter blocks — **exception**: facade package, not atom re-exports |
318+
| `@audio/eq-*` | Equalization & tone shaping (shelves, parametric, graphic) |
319+
| `@audio/weighting-*` | Frequency weighting curves (A, C, K, ITU468, RIAA) |
320+
| `@audio/auditory-*` | Auditory filterbanks (bark, ERB, mel, gammatone) |
321+
| `@audio/effect-*` | Audio effects (modulation, delay, distortion, utility) |
322+
| `@audio/dynamics-*` | Dynamics processing (compressor, limiter, gate, expander, transient, envelope) |
323+
| `@audio/spatial-*` | Spatial/channel utilities (midside, channelsplit, delay, crossfeed, panner, widen, haas) |
324+
| `@audio/speech-*` | Speech processing (formant, LPC, vocoder) |
325+
| `@audio/denoise-*` | Noise reduction & restoration |
326+
| `@audio/synth-*` | Synthesis (oscillator, pluck, rhythm, chirp, DTMF, risset) |
327+
| `@audio/spectral-*` | Spectral features (centroid, flatness, crest, rolloff, flux, slope, spread) |
328+
| `@audio/mir-*` | Music information retrieval (structure, transcribe, downbeat, chroma, chord, key, genre, mood, tags, fingerprint, similarity, drums, lyrics, separate, tonnetz, tempogram) |
329+
| `@audio/vocals-*` | Vocal isolation (isolate, remove) |
330+
331+
### What Stays in scijs
332+
333+
| Package | Why |
334+
|---------|-----|
335+
| `fourier-transform` | Generic FFT — used by radar, image processing, finance |
336+
| `window-function` | Generic window functions — spectral analysis primitive |
337+
| `digital-filter` | Generic filter design — electrical engineering, comms, control systems |
338+
| `periodic-function` | Generic waveform math — graphics, simulation, synthesis |
339+
340+
### Migration Path
341+
342+
1. **Phase 1**: Publish `@audio/*` atoms from current umbrella packages (no breaking changes to existing packages).
343+
2. **Phase 2**: Create `@audio/pitch`, `@audio/beat`, `@audio/shift`, `@audio/stretch`, `@audio/effect`, `@audio/denoise`, `@audio/dynamics`, `@audio/spatial`, `@audio/synth` umbrellas that re-export atoms.
344+
3. **Phase 3**: Create `@audio/filter` as audio-facing facade over `digital-filter`.
345+
4. **Phase 4**: Create `@audio/eq`, `@audio/weighting`, `@audio/auditory` umbrellas re-exporting their atoms.
346+
5. **Phase 5**: Optionally transfer `fourier-transform`, `window-function`, `digital-filter`, `periodic-function` to `scijs` org with deprecation notices in `audiojs`.

0 commit comments

Comments
 (0)