Skip to content

README.md

Latest commit

 

History

History
408 lines (332 loc) · 17.5 KB

README.md

File metadata and controls

408 lines (332 loc) · 17.5 KB
Raw

Clef

Clef is a local-first, open-source workspace that turns audio or video into an editable sheet-music draft. Upload a recording, follow the real processing stages, inspect the MusicXML in the browser, compare source audio with MIDI, fix notes, and export MusicXML, compressed MXL, MIDI, PDF, or an image when the required local renderer is available.

Automatic transcription has an honest ceiling. Clef produces a first draft, not finished engraving. Clean solo piano and other sparse pitched recordings work best. Dense mixes, rubato, ornamentation, unusual timbres, and noisy live recordings will need musical correction. The goal is to turn transcription from a blank-page task into a short editing pass.

Everything in the core path runs on the user's machine. No account, paid API, GPU, or API key is required.

What works

  • Audio and video ingest for MP3, WAV, FLAC, M4A, OGG, Opus, MP4, MOV, MKV, WebM, and AVI
  • 44.1 kHz playback and loudness-normalized analysis copies, with optional start/end trimming and a configurable 10-minute default cap
  • Piano, lead-sheet, multi-instrument, and automatic routing modes
  • Transkun, Basic Pitch, Demucs, Beat This!, torchcrepe, faster-whisper, Verovio, and yt-dlp in the default Python 3.11 quality install, plus optional MuseScore, OpenRouter, and Ollama adapters
  • A dependency-light spectral CPU transcriber and SciPy rhythm tracker when the optional ML packages are absent
  • Continuous beat mapping, per-beat straight/triplet subdivision selection, chord-cluster snapping, sensitivity cleanup, overlap repair, and moving two-hand piano split
  • Partitura engraving for suitable single-staff scores and music21 piano engraving with MusicXML 4.0 XSD validation
  • Browser notation with OpenSheetMusicDisplay
  • A waveform with beat/downbeat markers, source/MIDI A/B mix, and a current-time-derived score cursor that does not accumulate timing drift
  • Pitch nudges, octave nudges, deletion, title/composer/tempo/key/meter edits, dynamics, and moving/manual piano split controls using cached notes rather than rerunning a model
  • Lead-sheet chords, optional faster-whisper lyrics, and torchcrepe vocal-F0 refinement
  • SQLite job history, first-system thumbnails, and SQLite FTS5 library search
  • Original-audio and sampled Salamander Grand Piano playback, with a score cursor driven by source time rather than accumulated timers
  • Server-sent, measured stage updates rather than fake progress

URL import is shown only when yt-dlp is installed. PDF and image buttons are enabled only when the relevant local renderer produced the file.

Quick start

Requirements:

  • Python 3.11 or newer
  • Node.js 20 or newer (standard Node installers include npm)

macOS and Linux

git clone <your-clef-repository>
cd Clef
./run.sh

Windows PowerShell

git clone <your-clef-repository>
cd Clef
.\run.ps1

Open http://localhost:3000. The FastAPI service runs at http://127.0.0.1:8000; Next.js proxies /api to it.

The launchers create backend/.venv, install the core Python packages, install the frontend packages, and start both services. A system ffmpeg is preferred. When it is absent, the core install includes imageio-ffmpeg's local binary, so audio/video extraction still works.

The first startup is dependency installation and is therefore slower than later starts. Model downloads, when an optional model is installed, are cached under backend/.cache/.

Full quality tier and graceful fallback

run.sh and run.ps1 install the full packaged quality tier by default. They prefer Python 3.11 when it is available because that is the common runtime supported by every current model package. The public Beat This! checkpoint is downloaded once with a verified SHA-256; Transkun and Demucs weights download on their first use. All weights live under backend/.cache/.

To run the smaller core-only install, set CLEF_INSTALL_ML=0 in the shell:

# macOS / Linux
CLEF_INSTALL_ML=0 ./run.sh

# PowerShell
$env:CLEF_INSTALL_ML="0"; .\run.ps1

Every quality dependency remains runtime-optional. A failed wheel, import, weight download, separation, beat analysis, lyric pass, or renderer produces a visible job note and falls through instead of failing the core upload-to-MusicXML path. Basic Pitch's current upstream compatibility list ends at Python 3.11; on newer Python versions Clef omits it and uses another available adapter or the spectral CPU fallback.

Dependency Required? What it unlocks Behavior when absent
ffmpeg or imageio-ffmpeg Yes, bundled fallback Audio/video decoding, trimming, normalization Startup/API reports a clear install problem
Transkun v2 No Stronger piano performance transcription Basic Pitch or spectral CPU path
Basic Pitch No General pitched-instrument transcription Spectral CPU path
Beat This! No Neural beats and downbeats SciPy onset autocorrelation and phase inference
Demucs htdemucs No drums/bass/vocals/other stems Full-mix transcription with a visible note
torchcrepe No Median-filtered vocal F0 and octave/vibrato correction Basic Pitch melody remains usable
faster-whisper No Word timestamps and lyric alignment Lyrics omitted
MuseScore No Best PDF and PNG engraving Verovio, or MusicXML/MXL/MIDI only
Verovio No SVG and, where Cairo is present, PDF/PNG fallback MusicXML/MXL/MIDI only
yt-dlp No Rights-respecting URL ingest URL field hidden
OpenRouter key No Optional title/structure/practice extras Ollama or no extras
local Ollama No Key-free optional text extras No extras
Moss No Reserved semantic-library adapter SQLite FTS5 search

The Moss adapter is intentionally stubbed at the feature gate; the shipped, tested search implementation is SQLite FTS5. The app does not import or require a Moss SDK.

MuseScore executable names

Clef probes mscore, MuseScore4, musescore, and mscore4, plus the standard macOS MuseScore 4 application location and backend/.tools/MuseScore Studio.app. PDF/PNG regeneration is deferred until download after an edit so the interactive correction path stays fast.

Platform notes

  • macOS: Python from python.org or Homebrew and Node's LTS installer work. Apple Silicon users should prefer Python 3.11 when they want Basic Pitch.
  • Linux: install the Python development/runtime packages and Node 20+ from your distribution or NodeSource. A system ffmpeg is optional but recommended.
  • Windows: use 64-bit Python and Node, run run.ps1, and install MuseScore normally if PDF export is wanted. Some ML packages publish platform-specific wheels later than their macOS/Linux releases; Clef's CPU fallback remains available.

Configuration

Copy the example and keep credentials local:

cp .env.example .env.local

.env.local is ignored by Git. Never put keys in source files.

Variable Default Purpose
CLEF_MAX_DURATION_SECONDS 600 Maximum selected input duration
CLEF_DATA_DIR backend/data SQLite and per-job artifact directory
CLEF_DEMUCS_SEGMENT 7 Integer Demucs chunk size; htdemucs allows at most 7.8 s
CLEF_WHISPER_MODEL tiny.en Local faster-whisper lyric model
CLEF_TORCHCREPE_MODEL tiny Local vocal-F0 model (tiny or full)
OPENROUTER_API_KEY unset Enables optional text extras only
OPENROUTER_MODEL openrouter/free Optional OpenRouter model
OLLAMA_BASE_URL http://127.0.0.1:11434 Optional local Ollama
YTDLP_COOKIES_FILE unset Optional explicit yt-dlp cookies file
MOSS_ENABLED false Reserved adapter gate; FTS5 remains active

Pipeline

flowchart LR
    A["Upload or permitted URL"] --> B["ffmpeg ingest"]
    B --> C["Density / routing detection"]
    C --> D{"Demucs useful and available?"}
    D -->|yes| E["Cached stems"]
    D -->|no| F["Full mix"]
    E --> G["Transkun / Basic Pitch / CPU fallback"]
    F --> G
    B --> H["Beat This! or onset fallback"]
    G --> I["Performance notes"]
    H --> J["Beat, downbeat, tempo, meter, key maps"]
    I --> K["Per-beat quantization and cleanup"]
    J --> K
    K --> L["Hand split, voices, spelling, notation"]
    L --> M["MusicXML 4.0 XSD validation"]
    M --> N["OSMD view and downloads"]
    K --> O["Cached correction path"]
    O --> L
Loading

1. Ingest

The source is decoded to a 44.1 kHz stereo WAV for playback and a separate 44.1 kHz mono, loudness-normalized WAV for analysis. Video is passed with -vn, explicitly extracting only its audio track. A selected time range is applied before decoding.

2. Separation and routing

Sparse material or the explicit “solo instrument” control avoids unnecessary separation. Otherwise, an installed Demucs uses htdemucs with configurable integer segment chunking. Piano/full-mix routes prefer Transkun; vocals, bass, guitar/other, and other pitched stems use Basic Pitch; drums use onset and spectral-band classification in multi-instrument mode. Lead vocals can then pass through torchcrepe and faster-whisper.

3. Rhythm and key

The live Beat This! adapter uses File2Beats(checkpoint_path="final0", device="cpu", dbn=False). The fallback derives spectral flux, searches tempo lags by autocorrelation, aligns the beat phase to onsets, and chooses an accented bar phase. Tempo changes are emitted only after roughly 8% movement. Key starts with Krumhansl–Schmuckler chroma correlation and is cross-checked with music21's MIDI-note analysis.

4. Quantization

Seconds are converted to continuous beat positions by piecewise-linear interpolation. Notes are clustered within 50 ms, then each beat chooses one straight or triplet subdivision by timing error plus notation-complexity cost. The cleanup pass applies sensitivity-aware duration/velocity filtering, repairs small gaps and overlaps, and assigns piano hands with a windowed, hysteretic pitch split instead of a fixed middle C.

5. Engraving and validation

Partitura's current load_score_midi(..., quantization_unit=...) and save_musicxml(...) path is used for suitable single-staff targets. Piano uses music21 so the moving two-staff assignment is retained. The result is validated offline against the official MusicXML 4.0 XSD bundled by the musicxml package, then independently parsed by music21. An invalid Partitura result falls back to music21.

6. Rendering and correction

OpenSheetMusicDisplay loads the MusicXML string and renders SVG in the browser. MuseScore is preferred for local PDF/PNG; Verovio is the fallback. Raw performance notes, rhythm data, and score notes are cached per job, so sensitivity, key, meter, tempo, metadata, and note edits skip ingest, separation, and transcription. The first draft receives full MusicXML 4.0 XSD and independent music21 validation; corrections reuse the same validated writer and run a fast structural check.

API

The browser uses the same endpoints available to local integrations:

Method Path Purpose
GET /api/health Service health and ffmpeg readiness
GET /api/capabilities Runtime feature detection and UI notes
POST /api/jobs Multipart media upload and options
POST /api/jobs/url Optional yt-dlp URL job
GET /api/jobs Persistent library and optional FTS query
GET /api/jobs/{id}/events SSE stage stream
POST /api/jobs/{id}/reengrave Cached sensitivity/metadata update
POST /api/jobs/{id}/corrections Cached note correction
GET /api/jobs/{id}/files/{kind} MusicXML, MXL, MIDI, PDF, PNG, SVG, or audio

Evaluation

Run the complete synthesized harness:

cd backend
.venv/bin/python -m app.eval.harness --with-real-sample

It synthesizes known MIDI cases to PCM audio, runs Basic Pitch through ingest → transcription → rhythm → quantization → engraving → MusicXML validation, evaluates with mir_eval.transcription at 50 ms onset tolerance, verifies triplet and 3/4 notation, builds and extracts an MP4, and processes a public-domain acoustic-piano recording through Transkun. Synthetic additive oscillators intentionally use the general-instrument Basic Pitch path; they are not valid Transkun piano-timbre fixtures. Results are written to backend/eval_results.json.

Measured on macOS arm64, Python 3.11.4, CPU-only, with the full local model stack installed:

Case Precision Recall Onset F1 Onset + offset F1 Rhythm accuracy
C-major scale 1.000 0.875 0.933 0.933 0.875
Chromatic run 0.000 0.000 0.000 0.000 0.000
Block chords 1.000 0.833 0.909 0.909 0.833
Triplet passage 1.000 0.917 0.957 0.957 0.917
Tempo-change oscillator 0.000 0.000 0.000 0.000 0.000
3/4 passage 1.000 0.889 0.941 0.941 0.889

Additional observed checks:

  • Generated MP4 audio extraction: passed, 4.365 seconds
  • 3/4 MusicXML contains a 3/4 time signature
  • Triplet MusicXML contains <time-modification> with 3 actual notes
  • Public-domain acoustic piano: passed, 9.98 seconds, 8 Transkun notes, Beat This! tempo 115.38 BPM, valid MusicXML, and MuseScore round-trip
  • Dense real mix: 12 seconds, Demucs produced drums/bass/vocals/other, Basic Pitch plus approximate drums produced 87 raw events, Beat This! completed
  • torchcrepe vocal refinement: passed in 16.37 seconds; faster-whisper completed in 8.4 seconds and correctly returned no words for that non-vocal excerpt
  • Cached correction: 25 ms warm in the browser; measured backend key/meter updates were 575–631 ms. The first-ever music21 import took 3.34 seconds.
  • Two-minute playback: at 1:58 the source-derived cursor was on the final notation system; it reached 2:00 without drift or console errors
  • Generated MP4 extraction, 3/4 signature, triplet notation, all export formats, production frontend build, core-fallback tests, and single-command startup all passed

The zero scores are retained deliberately: these short, immediate oscillator runs fall below Basic Pitch's current minimum-note behavior. Synthetic numbers verify structural behavior and are not a claim about accuracy on real music.

Tests and production build

backend/.venv/bin/pytest

cd frontend
npm run build

Upstream API notes

The adapters were implemented against current upstream documentation:

  • Transkun documents transkun input.mp3 output.mid --device cuda and a CPU default.
  • Basic Pitch returns (model_output, midi_data, note_events) from predict(...).
  • Beat This! exposes beat_this.inference.File2Beats; Clef deliberately keeps dbn=False.
  • Partitura supports load_score_midi with a quantization unit and save_musicxml.
  • music21 handles MIDI/MusicXML parsing, notation cleanup, and export.
  • Demucs documents htdemucs, --segment, and the four standard stems. The installed Demucs 4.1 CLI parses --segment as an integer, so Clef uses 7.
  • OpenSheetMusicDisplay accepts an XML string through async load(...), followed by render().
  • The official MusicXML 4.0 XSD is the validation contract.

One live-doc difference worth calling out: Partitura's own bundled validator still references MusicXML 3.1, so Clef uses the separate MusicXML 4.0 schema for the returned artifact.

Model and code licenses

Clef itself is MIT licensed. Optional upstream packages and weights retain their own licenses:

Model / component License
Transkun code and shipped checkpoint MIT
Spotify Basic Pitch code/model Apache 2.0
CPJKU Beat This! code/model weights MIT
Demucs code/models MIT; users should review upstream training-data notes
torchcrepe / converted CREPE weights MIT
faster-whisper code MIT; selected Whisper model terms also apply
Partitura Apache 2.0
music21 BSD-3-Clause
OpenSheetMusicDisplay BSD-3-Clause
Verovio LGPL-3.0
MuseScore Studio GPL-3.0
Salamander Grand Piano samples CC BY 3.0, Alexander Holm

See each linked upstream project before redistributing its model weights. Clef does not commit third-party weights.

Input rights

You are responsible for using only recordings and URLs you have the right to download, process, transcribe, and export. Clef does not grant rights to music, performances, recordings, streams, or derived scores. URL support is a local tool, not a license or a way around a service's terms.

Data and privacy

Job records, uploads, normalized audio, cached notes, and exports live under backend/data/ by default. Delete that directory to remove the local library. The optional OpenRouter integration is never used in transcription; enabling it can send only a compact score summary for optional text extras. Source audio is not sent to an LLM.

Project layout

.
├── run.sh / run.ps1
├── backend
│   ├── app
│   │   ├── main.py
│   │   ├── jobs.py
│   │   ├── pipeline/
│   │   ├── integrations/
│   │   └── eval/harness.py
│   ├── requirements.txt
│   └── eval_results.json
├── frontend
│   ├── app/
│   └── components/
└── samples/