From 453122067f84ffe2c15cca4a722a04bca1dd162d Mon Sep 17 00:00:00 2001 From: Jaewan Park Date: Sat, 13 Jun 2026 19:04:22 +0900 Subject: [PATCH] docs: plan arrangement clip placement playback --- PLANS.md | 9 +- README.md | 7 +- docs/architecture.md | 5 + docs/audio-engine.md | 20 +++ docs/data-model.md | 43 +++++ ...arrangement-clip-placement-and-playback.md | 166 ++++++++++++++++++ docs/product.md | 4 +- docs/testing.md | 8 + docs/ui-design.md | 18 ++ 9 files changed, 273 insertions(+), 7 deletions(-) create mode 100644 docs/features/16-arrangement-clip-placement-and-playback.md diff --git a/PLANS.md b/PLANS.md index 4a2a9b9..73234ab 100644 --- a/PLANS.md +++ b/PLANS.md @@ -30,6 +30,7 @@ M1 should include: Use this section as the current execution order for agent work. Feature document numbering describes the planned product sequence, but actual issue order may change as dependencies, review feedback, or implementation risks become clearer. 1. #39 WAV file import as audio clip -> `docs/features/15-wav-file-import-as-audio-clip.md` +2. #41 Arrangement clip placement and playback -> `docs/features/16-arrangement-clip-placement-and-playback.md` ## Planned Milestones @@ -48,10 +49,10 @@ Use this section as the current execution order for agent work. Feature document 13. Arrangement mixer panel UI shell. 14. Hybrid clip loop playback. 15. WAV file import as audio clip. -16. Project export/import. -17. IndexedDB autosave. -18. Arrangement data model. -19. Arrangement editing and playback. +16. Arrangement clip placement and playback. +17. Project export/import. +18. IndexedDB autosave. +19. Arrangement data model refinements. 20. Mixer audio routing. 21. Basic effects. diff --git a/README.md b/README.md index 01c5236..e02e48f 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ Build a practical clip-based music creation tool that can grow beyond a demo whi ## Current Status -The project has an initial Vite + React + TypeScript scaffold. DAW features such as audio playback, sequencing, piano roll editing, and arrangement editing have not been implemented yet. +The project has an initial Vite + React + TypeScript scaffold and early DAW UI/audio features under active development. Arrangement placement, arrangement playback, persistence, and broader song-building workflows are still planned work. ## Getting Started @@ -67,6 +67,7 @@ Feature task specs: - [13 Drum Step Subdivisions](docs/features/13-drum-step-subdivisions.md) - [14 Arrangement Mixer Panel UI Shell](docs/features/14-arrangement-mixer-panel-ui-shell.md) - [15 WAV File Import as Audio Clip](docs/features/15-wav-file-import-as-audio-clip.md) +- [16 Arrangement Clip Placement and Playback](docs/features/16-arrangement-clip-placement-and-playback.md) ## Initial Roadmap @@ -76,7 +77,9 @@ Feature task specs: 4. Build a 16-step drum sequencer. 5. Build a basic 1-bar piano roll. 6. Combine drum and note events into hybrid clip loop playback. -7. Add project export/import and persistence features. +7. Import local WAV files as audio clips. +8. Place clips on the arrangement timeline and play the arranged song. +9. Add project export/import and persistence features. ## Notes diff --git a/docs/architecture.md b/docs/architecture.md index 89a1bb4..d0262e0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,6 +14,7 @@ The main rule is separation of concerns: UI rendering, persistence, and audio sc - Dispatch user actions to state/model logic. - Display audio state such as stopped, playing, paused, and playhead position. - Render song-level UI shells such as arrangement and mixer panels. +- Let users place, move, select, and delete arrangement clip instances. - Use `requestAnimationFrame` for visual playheads where needed. - Avoid owning exact audio timing. @@ -22,6 +23,7 @@ The main rule is separation of concerns: UI rendering, persistence, and audio sc - Define serializable project data types. - Store musical time in ticks. - Provide pure transformations for creating, editing, duplicating, and deleting clips and events. +- Provide pure transformations for creating, moving, and deleting arrangement clip instances. - Avoid references to Web Audio runtime objects. ### Audio engine @@ -30,6 +32,7 @@ The main rule is separation of concerns: UI rendering, persistence, and audio sc - Load and decode sample assets. - Store runtime-only decoded sample data. - Schedule audio against `AudioContext.currentTime`. +- Schedule arrangement playback from placed clip instances when `SONG` mode is active. - Later, own mixer routing, track gain nodes, meters, and effect nodes when those features exist. - Expose a small typed API to the UI and feature code. - Never depend on React components. @@ -82,6 +85,8 @@ Imported browser files are also runtime or persistence-layer data. Project JSON Future mixer runtime data, including `GainNode`, `AnalyserNode`, effect nodes, meter buffers, and active routing graphs, also belongs to the audio engine runtime and must not be stored in project JSON. +Arrangement placement data is serializable. A placed clip should be represented by a `ClipInstance` with stable IDs, `startTick`, `lengthTicks`, and track membership. Drag state, pointer coordinates, DOM measurements, scheduler timers, decoded buffers, and active audio nodes are runtime-only and must not be persisted. + ## Future Desktop Packaging Considerations Keep the app browser-first, but avoid assumptions that block future Electron or Tauri packaging: diff --git a/docs/audio-engine.md b/docs/audio-engine.md index 7f2399e..80a38c1 100644 --- a/docs/audio-engine.md +++ b/docs/audio-engine.md @@ -51,6 +51,26 @@ The first imported WAV implementation may support a simple preview or clip playb Imported audio clip duration is source media duration in seconds. Future arrangement placement should convert arrangement positions and visible instance lengths to ticks, while source offsets remain sample-local seconds. Without a time-stretching feature, resizing an audio clip instance should trim/crop playback or extend silence rather than stretch the audio to a new musical duration. +## Arrangement Playback + +`PAT` mode playback targets the selected clip editor. `SONG` mode playback should target placed `ClipInstance` objects on the arrangement timeline. + +Arrangement playback should: + +- Read serializable clip instances from project or app model state. +- Convert arrangement tick positions to `AudioContext.currentTime` scheduling times. +- Expand hybrid clip drum events to `clipInstance.startTick + drumEvent.startTick`. +- Expand hybrid clip note events to `clipInstance.startTick + noteEvent.startTick`. +- Schedule audio clips at `clipInstance.startTick` when their runtime sample data is available. +- Respect `clipInstance.lengthTicks` as the visible and playable duration boundary. +- Keep play, pause, resume, and stop behavior separate from React render timing. + +The first arrangement playback implementation may play linearly from tick 0 through the end of the last placed clip instance, then stop. Arrangement loop regions can be added later. + +Imported audio clips should play at original speed unless a later time-stretching feature explicitly changes that behavior. If an audio clip instance is shorter than the source buffer, playback should be cropped. If the instance is longer than the source buffer, playback may end naturally and leave silence. If runtime file data is missing after refresh, the engine should report a clear missing-source error rather than silently failing. + +Arrangement playback still uses the lookahead scheduler. UI drag state, arrangement DOM geometry, visual playheads, decoded buffers, and active source nodes remain runtime-only. + ## One-shot Sample Playback Use a new `AudioBufferSourceNode` for every one-shot playback. A source node cannot be restarted after it has played. diff --git a/docs/data-model.md b/docs/data-model.md index f45b8f0..b42276e 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -130,6 +130,48 @@ export interface AudioClip { Future arrangement resizing should be non-destructive. The arrangement should store resize/trim decisions on `ClipInstance`, for example `lengthTicks` and optional `sourceOffsetSeconds`, instead of modifying the source audio clip or embedded file. Without a dedicated time-stretching feature, resizing an imported audio clip instance should mean trimming/cropping playback or showing silence after the source ends; it should not imply tempo-matched stretching. +## Arrangement Clip Placement + +The arrangement view places reusable clips on tracks using `ClipInstance` objects. + +`Clip` owns reusable source content: + +- Hybrid clip drum and note events. +- Audio clip source metadata. +- Clip name and source identity. + +`ClipInstance` owns song placement: + +- Which source clip is placed. +- Which track contains it. +- Where it starts in arrangement ticks. +- How long the placed instance lasts in arrangement ticks. +- Optional source offset for audio clips. + +Illustrative shape: + +```ts +export interface ClipInstance { + id: string; + clipId: string; + trackId: string; + startTick: Tick; + lengthTicks: Tick; + sourceOffsetSeconds?: number; +} +``` + +The first arrangement placement feature should create, move, select, and delete `ClipInstance` objects without mutating the source `Clip`. Deleting a placed clip from the arrangement removes only that instance. It does not delete the sidebar clip. + +Default instance lengths: + +- Hybrid clip: use the source clip's `lengthTicks`, initially 1920 ticks for a 1-bar clip. +- Audio clip: derive an initial `lengthTicks` from `durationSeconds` and current `tempoBpm` when placed, or use an equivalent helper that keeps arrangement placement tick-based. + +Without time stretching, imported audio playback runs at original speed. If an audio clip instance is shorter than the source, playback is cropped. If it is longer than the source, playback may end naturally and leave silence. Changing project tempo can change the musical grid without changing the underlying audio source speed until a later time-stretching feature exists. + +Snap and movement should update tick values, not pixel positions. UI geometry is derived from `startTick`, `lengthTicks`, track order, and timeline constants. + ## Bundled Drum Sample Naming and Display Bundled drum sample files live under `public/samples/drums/`. @@ -366,6 +408,7 @@ export interface ClipInstance { trackId: string; startTick: Tick; lengthTicks: Tick; + sourceOffsetSeconds?: number; } export interface DrumEvent { diff --git a/docs/features/16-arrangement-clip-placement-and-playback.md b/docs/features/16-arrangement-clip-placement-and-playback.md new file mode 100644 index 0000000..4df8ecc --- /dev/null +++ b/docs/features/16-arrangement-clip-placement-and-playback.md @@ -0,0 +1,166 @@ +# Feature: 16 Arrangement Clip Placement and Playback + +## Status +Planned + +## Goal + +Allow users to place reusable clips on the `SONG` arrangement timeline and play the arranged song from the transport. + +This turns the arrangement view from a static shell into the first usable song-building surface. + +## Context + +The app already has: + +- A `PAT` mode for focused clip editing. +- A `SONG` mode arrangement UI shell. +- A left sidebar that manages clips and clip child items. +- Transport play, pause, resume, stop, playhead, and tempo behavior for clip playback. +- Planned WAV import as audio clips in `docs/features/15-wav-file-import-as-audio-clip.md`. + +Users now need to place clips horizontally in time and vertically across tracks. Playback in `SONG` mode should schedule the placed clip instances instead of looping only the selected `PAT` clip. + +## Scope + +Included: +- Make sidebar clips draggable into the arrangement timeline where practical. +- Create serializable `ClipInstance` entries when clips are placed. +- Store arrangement placement with `clipId`, `trackId`, `startTick`, and `lengthTicks`. +- Render arrangement clip blocks from project or app model state instead of static demo blocks. +- Move existing clip instances horizontally and vertically by drag where practical. +- Snap dropped and moved clip instances to the arrangement grid. +- Select and delete placed clip instances. +- Keep the existing arrangement track list and timeline layout aligned. +- Add a visual arrangement playhead in `SONG` mode. +- Make transport play, pause, resume, and stop operate on arrangement playback when `SONG` mode is active. +- Schedule placed hybrid clips by expanding their drum and note events relative to each `ClipInstance.startTick`. +- Schedule placed audio clips when imported audio runtime data is available. +- Keep imported audio playback at original speed; no time stretching in this feature. +- Add focused tests for clip instance creation, moving, deletion, snapping, and scheduler event expansion where practical. +- Update relevant product, architecture, data model, audio, UI, and testing docs. + +Excluded: +- Time stretching or tempo matching imported audio. +- Clip instance resize handles. +- Clip splitting. +- Arrangement recording. +- Track creation, deletion, or reordering. +- Multi-select, copy/paste, duplicate shortcuts, or marquee selection. +- Arrangement loop region editing. +- Persistent project export/import beyond serializable in-memory model changes. +- IndexedDB persistence for imported audio bytes. +- Full mixer audio routing, real level metering, or effects. + +## Constraints + +- React UI must not own exact audio timing. +- Store arrangement positions and lengths in ticks. +- Use PPQ 480, where one 4/4 bar is 1920 ticks. +- Keep project data serializable. +- Do not store `AudioBuffer`, `AudioNode`, `File`, `Blob`, object URLs, scheduler timers, or drag runtime objects in project JSON. +- Playback must schedule audio against `AudioContext.currentTime`. +- Audio clip source duration may be seconds because it describes media length, but arrangement placement still uses ticks. +- Imported audio clips must fail clearly if their runtime file data is missing. +- Use CSS Modules and semantic design tokens for UI changes. +- Preserve existing `PAT` mode clip editing behavior. + +## Data Model Notes + +The arrangement should use serializable clip instances: + +```ts +export interface ClipInstance { + id: string; + clipId: string; + trackId: string; + startTick: Tick; + lengthTicks: Tick; + sourceOffsetSeconds?: number; +} +``` + +For hybrid clips, the first implementation should use the clip's `lengthTicks` as the default instance length. Hybrid clip events play once relative to the instance start. Repeating a pattern should be represented by multiple placed instances until a later feature adds looping or repeat handles. + +For imported audio clips, `lengthTicks` controls the visible and scheduled clip-instance duration. Without time stretching, playback uses the decoded audio at original speed: + +- If the instance is shorter than the source, playback is cropped. +- If the instance is longer than the source, the remaining duration is silence. +- If runtime imported file data is unavailable after refresh, the clip instance should remain visible but audio playback should report a clear missing-source limitation. + +The first implementation may seed a small set of arrangement tracks in app state. Track creation, deletion, and reordering are separate tasks. + +## UI Notes + +Use the existing `SONG` arrangement layout: + +- Sidebar clip rows are drag sources. +- Arrangement track lanes are drop targets. +- Placed clips render as timeline blocks. +- Dragging a placed clip moves it to another snapped start time or track. +- Selecting a placed clip shows a clear selected state. +- Deleting a placed clip instance removes only that arrangement placement, not the source clip from the sidebar. + +Snap should start simple. Prefer beat-level snapping, using 480 ticks, unless the existing arrangement snap control already supports a narrower value. Later features may add user-selectable snap resolution. + +If browser drag-and-drop is awkward for track timeline geometry, pointer-based dragging is acceptable. The feature should still behave like drag placement from the user's perspective. + +## Audio Notes + +`SONG` playback should schedule arrangement events, not selected-clip loop events. + +The audio engine or feature orchestration should expand clip instances into runtime scheduler events: + +- Hybrid clip drum events: `instance.startTick + drumEvent.startTick`. +- Hybrid clip note events: `instance.startTick + noteEvent.startTick`. +- Audio clips: source starts at `instance.startTick`, optionally from `sourceOffsetSeconds`. + +The arrangement transport may initially play linearly from tick 0 through the end of the last clip instance and then stop. Arrangement loop ranges can be added later. + +Events already scheduled inside the lookahead window may still play briefly after edits, pause, or stop. Keep the scheduling window short enough for interactive editing. + +## Done when + +- Users can drag a sidebar clip into a `SONG` arrangement track. +- A placed clip block appears at the snapped time and correct track. +- Placed clip blocks are backed by serializable `ClipInstance` data. +- Static demo arrangement clips are removed or replaced by seeded serializable model state. +- Users can move a placed clip instance to another snapped time or track. +- Users can select and delete a placed clip instance without deleting the source clip. +- `SONG` transport playback schedules placed clip instances. +- Pause, resume, and stop work in `SONG` mode. +- A visual arrangement playhead follows playback and resets or hides appropriately when stopped. +- Hybrid clip drum and note events play at their arrangement positions. +- Audio clip instances play when runtime imported audio data is available. +- Missing imported audio runtime data is reported clearly instead of silently failing. +- `PAT` mode clip editing and clip playback still work. +- Relevant tests and docs are updated. + +## Verification + +Run: +- `npm run typecheck` +- `npm run lint` +- `npm run test` +- `npm run build` + +Manual check: +- Switch to `SONG` mode. +- Drag a sidebar clip into an arrangement track. +- Confirm the placed clip aligns to the snapped grid. +- Drag the placed clip to another time and track. +- Select and delete the placed clip. +- Place multiple clips across tracks and press play. +- Confirm the arrangement playhead roughly matches audible playback. +- Confirm pause, resume, and stop behavior in `SONG` mode. +- Confirm `PAT` mode still edits and plays the selected clip. +- If WAV import is implemented, place an imported audio clip and confirm it plays when runtime data is available. +- Refresh the page and confirm missing imported audio data is handled clearly if persistence is not implemented. + +## PR notes + +- Reference issue #41. +- Explain the `ClipInstance` model changes or model usage. +- Explain how sidebar drag placement and arrangement move/delete work. +- Explain how `SONG` playback differs from `PAT` clip loop playback. +- State clearly that time stretching, resizing, arrangement loop regions, and persistent imported audio bytes are deferred. diff --git a/docs/product.md b/docs/product.md index 3e4e9e8..6704f19 100644 --- a/docs/product.md +++ b/docs/product.md @@ -16,7 +16,8 @@ Build a browser-first mini DAW for creating electronic music with short clips. T 2. Create or select a 1-bar clip. 3. Build a clip with drum events and pitched notes, or import a local WAV file as an audio clip. 4. Start loop playback and edit while listening. -5. Later, arrange clips into a larger song. +5. Place clips on the `SONG` arrangement timeline. +6. Play the arranged timeline to build a larger song. ## MVP Definition @@ -44,6 +45,7 @@ The MVP is a browser-first 1-bar hybrid clip editor with: - Users can create a short 1-bar loop with drums and pitched notes. - Playback timing is stable enough for simple electronic music loops. +- Users can place clips on an arrangement timeline and hear the placed clips in song order. - Project data can be represented as JSON without runtime audio objects. - The codebase separates UI rendering, project state, persistence, and audio scheduling. - Future contributors can pick up feature specs and implement small, reviewable tasks. diff --git a/docs/testing.md b/docs/testing.md index 94c8845..d3e6017 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -20,6 +20,8 @@ CI uses `--if-present` while the repository is still before the Vite scaffold. - Data model transformations: unit tests. - Drum step subdivision tick math and event toggling: unit tests. - Clip collection and sidebar membership transformations: unit tests. +- Arrangement clip instance creation, movement, deletion, and snapping: unit tests. +- Arrangement scheduler event expansion from clip instances: unit tests where practical. - Pitched instrument metadata and sample-zone mapping: unit tests. - Tempo control and scheduler tempo update behavior: unit tests where practical. - Sustain loop point calculations: unit tests. @@ -61,6 +63,10 @@ tests/unit/utils/tick-time.test.ts - Clip add/delete/rename selection fallback. - Per-clip pitched instrument add/delete behavior. - Removing pitched instruments that own note events. +- Arrangement clip placement snapping. +- Arrangement clip move/delete behavior. +- Arrangement playback event expansion across clip instance offsets. +- Arrangement playhead behavior during play, pause, resume, and stop. - Sample start offsets, optional sustain loop points, and note release behavior. - Sampler sustain fallback behavior when loop metadata is missing or invalid. - Clip duplication. @@ -93,6 +99,8 @@ Manual audio checks should verify: - Mixer UI shell checks should verify fader, mute, solo, meter placeholder, and effect slot visuals without implying real audio routing. - Drum subdivision settings of `1`, `2`, and `3` should toggle and play hits at the expected rhythmic positions. - WAV import checks should verify valid WAV import, invalid file rejection, imported clip selection, displayed duration metadata, and clear behavior after refresh when imported file persistence is not implemented. +- Arrangement placement checks should verify dragging clips into tracks, moving placed clips, deleting placed clips, and playback from `SONG` mode. +- Imported audio clip arrangement checks should verify clear missing-source behavior after refresh until imported file persistence exists. Use headphones or speakers at a safe volume. Record browser, OS, and device details when reporting audio timing issues. diff --git a/docs/ui-design.md b/docs/ui-design.md index 03bb758..f3ddd78 100644 --- a/docs/ui-design.md +++ b/docs/ui-design.md @@ -113,6 +113,24 @@ Track headers, timeline lanes, and clip blocks must align exactly. Use shared ge Do not introduce real arrangement data, persisted clip instances, arrangement playback, or drag-and-drop editing in the first arrangement UI shell unless a feature spec explicitly expands the scope. +## Arrangement Clip Placement + +The next arrangement step should make the `SONG` view editable and playable. + +- Sidebar clips act as drag sources. +- Arrangement track lanes act as drop targets. +- Dropping a clip creates a visible clip block backed by a serializable `ClipInstance`. +- Moving a clip block updates `startTick` and `trackId`, not stored pixel positions. +- Selecting a clip block should show a clear selected state. +- Deleting a selected clip block removes only the arrangement instance, not the source clip in the sidebar. +- Static demo clip blocks should be removed or replaced with seeded serializable state. + +Use shared arrangement geometry constants for row height, ruler height, bar width, beat width, timeline width, and track header width. Inline styles are acceptable for dynamic clip geometry such as `left`, `top`, `width`, and transform values. + +Start with a simple snap policy, such as beat-level snapping at 480 ticks, unless a feature spec introduces user-selectable snap values. + +`SONG` mode playback should render a visual arrangement playhead. The playhead may use `requestAnimationFrame` for display, but exact playback must remain scheduled by the audio engine against `AudioContext.currentTime`. + ## Arrangement Mixer Panel The mixer should start as a bottom dock inside `SONG` mode, below the arrangement timeline.