Merge pull request #15 from mondain/feature/ctedash

Add CTE LL-DASH ingest publisher

Author: mondain

CMakeLists.txt

@@ -149,6 +149,7 @@ add_library(openmoq_publisher_lib STATIC
     src/cmsf_packager.cpp
     src/live_srt_ingest.cpp
     src/live_srt_config.cpp
+    src/live_dash_ingest.cpp
     src/moq_draft.cpp
     src/mp4_box.cpp
     src/publisher_api.cpp
@@ -235,6 +236,12 @@ if(OPENMOQ_BUILD_TESTS)
     target_link_libraries(openmoq-publisher-live-srt-config-tests PRIVATE openmoq_publisher_lib)
     add_test(NAME openmoq-publisher-live-srt-config-tests COMMAND openmoq-publisher-live-srt-config-tests)
 
+    add_executable(openmoq-publisher-live-dash-tests
+        tests/live_dash_ingest_test.cpp
+    )
+    target_link_libraries(openmoq-publisher-live-dash-tests PRIVATE openmoq_publisher_lib)
+    add_test(NAME openmoq-publisher-live-dash-tests COMMAND openmoq-publisher-live-dash-tests)
+
     add_executable(openmoq-publisher-transport-tests
         tests/moqt_session_test.cpp
     )

README.md

@@ -14,8 +14,9 @@ It turns MP4 input into CMSF-style publishable objects, builds draft-aware MOQT
 - Preserves HEVC signaling and normalizes `hev1` to `hvc1` when needed.
 - Builds publish plans with catalog, optional MSF media timeline and SAP event timeline metadata, and media objects.
 - Emits generated objects and catalog metadata to disk for inspection.
-- Supports draft-aware MOQT framing for drafts 14, 16, and 18.
+- Supports draft-aware MOQT framing for drafts 14, 16, 17, and 18.
 - Publishes over Raw QUIC or WebTransport when picoquic and picotls are available.
+- Accepts live CTE LL-DASH/CMAF ingest over HTTP/1.1 chunked `POST` or `PUT` requests.
 
 ## Quick Start
 
@@ -53,7 +54,7 @@ OPENMOQ_PICOQUIC_TRACE=1 ./build/openmoq-publisher \
   --paced
 ```
 
-Live ingest examples (choose one path, not both):
+Live ingest examples (choose one path):
 
 1. SRT ingest path (`--live-source srt`)
 
@@ -136,9 +137,58 @@ ffmpeg -i /home/ubuntu/bbb_sunflower_1080p_30fps_normal.mp4 \
     --forward 0
 ```
 
+3. CTE LL-DASH ingest path (`--live-source dash`)
+
+Start the publisher with an HTTP/1.1 chunked CMAF ingest listener and a MoQ relay target:
+
+```bash
+./build/openmoq-publisher \
+  --live-source dash \
+  --dash-listen 0.0.0.0:8080 \
+  --dash-path /ingest \
+  --endpoint https://127.0.0.1:4433/moq \
+  --transport webtransport \
+  --namespace live \
+  --draft 18 \
+  --publish-catalog \
+  --forward 1 \
+  --insecure
+```
+
+Send CMAF/fMP4 bytes with chunked transfer encoding. Multiple concurrent paths under the prefix are accepted, for example `/ingest/video` and `/ingest/audio`; each path gets path-prefixed track names in the MoQ catalog.
+
+```bash
+curl -X PUT \
+  -H 'Transfer-Encoding: chunked' \
+  -H 'Content-Type: video/iso.segment' \
+  --data-binary @live-video.cmaf \
+  http://127.0.0.1:8080/ingest/video
+```
+
+FFmpeg can push live DASH/CMAF requests directly to the ingest prefix. This example creates two video representations plus audio; FFmpeg writes representation requests such as `/ingest/video0`, `/ingest/video1`, and `/ingest/video2`.
+
+```bash
+ffmpeg -re \
+  -f lavfi -i "testsrc2=size=1280x720:rate=25" \
+  -f lavfi -i "anullsrc=r=48000:cl=stereo" \
+  -filter_complex "[0:v]split=2[v1][v2];[v1]scale=1280:720[v720];[v2]scale=640:360[v360]" \
+  -map "[v720]" -c:v:0 libx264 -b:v:0 1500k -g 50 -keyint_min 50 -sc_threshold 0 \
+  -map "[v360]" -c:v:1 libx264 -b:v:1 500k -g 50 -keyint_min 50 -sc_threshold 0 \
+  -map 1:a -c:a aac -b:a 128k \
+  -f dash -seg_duration 2 -use_template 1 -use_timeline 0 \
+  -init_seg_name 'video$RepresentationID$' \
+  -media_seg_name 'video$RepresentationID$' \
+  -adaptation_sets "id=0,streams=v id=1,streams=a" \
+  -multiple_requests 1 -streaming 1 -remove_at_exit 0 \
+  -window_size 20 -extra_window_size 20 \
+  http://127.0.0.1:8080/ingest/
+```
+
+Use `--forward 1` when the relay should receive objects immediately. Use `--forward 0` for await-subscribe mode, where media is sent after the relay forwards subscriber interest for the published tracks. A printed `connection_id=` confirms transport and MOQT setup only; it does not confirm namespace acceptance or a downstream subscription.
+
 `--live-source both` is intentionally not supported.
 
-On Windows, replace `./build/openmoq-publisher` with `build\Release\openmoq-publisher.exe` or the matching build configuration path.
+On Windows, replace `./build/openmoq-publisher` with `build\Release\openmoq-publisher.exe` or the matching build configuration path. The DASH ingest listener itself is currently supported on Unix-like platforms; Windows builds report the mode as unsupported.
 
 ## Documentation
 
@@ -154,6 +204,7 @@ On Windows, replace `./build/openmoq-publisher` with `build\Release\openmoq-publ
 | CAT4MOQ auth example | [examples/auth/README.md](examples/auth/README.md) |
 | Protocol mapping | [docs/protocol-mapping.md](docs/protocol-mapping.md) |
 | WebTransport compliance | [docs/webtransport-compliance.md](docs/webtransport-compliance.md) |
+| macOS DASH shutdown behavior | [docs/macos-accept-shutdown-quirk.txt](docs/macos-accept-shutdown-quirk.txt) |
 | Transport plan | [docs/transport-plan.md](docs/transport-plan.md) |
 | Project status and roadmap | [docs/status.md](docs/status.md) |
 
@@ -171,6 +222,6 @@ Localized Publisher API guides are available in [Spanish](docs/publisher-api.es.
 
 ## Current Status
 
-The publisher can generate publish plans, emit inspectable output, and publish over picoquic-backed Raw QUIC and WebTransport transports. Draft 14 is the primary target, draft 16 is maintained as a compatibility profile, and draft 18 support is implemented for version selection, setup/request framing codec paths, and request-stream response correlation while interop hardening continues.
+The publisher can generate publish plans, emit inspectable output, and publish over picoquic-backed Raw QUIC and WebTransport transports. Draft 14 is the primary target, draft 16 is maintained as a compatibility profile, and drafts 17 and 18 provide the newer VI64 and request-stream protocol profiles. Draft-18 subscriber interest is accepted on fragmented request-stream reads and answered on the same request stream. The draft-19 text is archived under `docs/superpowers/specs/` for review only; `--draft 19` is not implemented.
 
 For the detailed roadmap, see [docs/status.md](docs/status.md).

docs/ctedash-implementation-plan.md

@@ -0,0 +1,248 @@
+# CTE LL-DASH Ingest Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add CTE LL-DASH as a live publishing mechanism by accepting multiple concurrent HTTP/1.1 chunked CMAF ingest paths and forwarding resulting live objects through the existing MoQ publisher.
+
+**Architecture:** Implement a small HTTP/1.1 ingest server that accepts `POST`/`PUT` requests with `Transfer-Encoding: chunked`, decodes request bodies incrementally, and feeds decoded bytes into a CMAF live adapter. The adapter reuses `StreamingMp4Reader`, `extract_tracks`, `build_live_catalog`, `build_live_fragment`, and `Publisher::publish_live_objects(...)`, keeping the new surface limited to one live ingest module and CLI wiring.
+
+**Tech Stack:** C++20, POSIX sockets on non-Windows builds, existing MP4/CMAF parser and MoQ publisher API, CMake tests.
+
+---
+
+## File Structure
+
+- Create `include/openmoq/publisher/live_dash_ingest.h`
+  - Public config for the DASH ingest listener.
+  - Queue-backed live object source builder.
+  - Testable chunked-transfer decoder and CMAF stream adapter.
+- Create `src/live_dash_ingest.cpp`
+  - HTTP request parser.
+  - Chunked transfer decoder.
+  - Multi-path connection handler.
+  - CMAF init/media object production.
+- Create `tests/live_dash_ingest_test.cpp`
+  - Unit tests for chunked decoding, CMAF object production, and concurrent path behavior.
+- Modify `include/openmoq/publisher/cli_options.h`
+  - Add `LiveSourceKind::kDash` and DASH listener options.
+- Modify `src/cli_options.cpp`
+  - Parse `--live-source dash`, `--dash-listen`, `--dash-path`, and `--dash-queue-depth`.
+- Modify `src/main.cpp`
+  - Start DASH ingest mode and call `Publisher::publish_live_objects(...)`.
+- Modify `CMakeLists.txt`
+  - Compile the new module and add the new test executable.
+- Modify `docs/quickstart.md` and `README.md`
+  - Document the new ingest mode after implementation.
+
+## Task 1: Plan and Baseline
+
+- [ ] **Step 1: Save this plan**
+
+Create `docs/ctedash-implementation-plan.md` with this content.
+
+- [ ] **Step 2: Verify baseline branch**
+
+Run:
+
+```bash
+git status -sb
+```
+
+Expected: branch is `feature/ctedash`; only this plan file is new before implementation starts.
+
+## Task 2: Chunked Transfer Decoder
+
+**Files:**
+- Create: `include/openmoq/publisher/live_dash_ingest.h`
+- Create: `src/live_dash_ingest.cpp`
+- Test: `tests/live_dash_ingest_test.cpp`
+- Modify: `CMakeLists.txt`
+
+- [ ] **Step 1: Write failing tests**
+
+Add tests that feed chunked bytes split across arbitrary input boundaries:
+
+```cpp
+ok &= expect_chunked_decodes("4\r\nWiki\r\n5\r\npedia\r\n0\r\n\r\n", "Wikipedia");
+ok &= expect_chunked_decodes("4;token=value\r\nWiki\r\n0\r\n\r\n", "Wiki");
+ok &= expect_chunked_rejects("FFFFFFFFFFFFFFFFF\r\nx\r\n0\r\n\r\n");
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run:
+
+```bash
+cmake --build build --target openmoq-publisher-live-dash-tests
+./build/openmoq-publisher-live-dash-tests
+```
+
+Expected: build or test fails because the new module does not exist yet.
+
+- [ ] **Step 3: Implement minimal decoder**
+
+Implement `ChunkedBodyDecoder::append(...)` and `ChunkedBodyDecoder::take_decoded()` with explicit states for size line, data, data CRLF, trailers, complete, and error. Enforce a configurable max chunk size and reject integer overflow.
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run the same test command. Expected: all chunked decoder tests pass.
+
+## Task 3: CMAF Live Object Adapter
+
+**Files:**
+- Modify: `include/openmoq/publisher/live_dash_ingest.h`
+- Modify: `src/live_dash_ingest.cpp`
+- Test: `tests/live_dash_ingest_test.cpp`
+
+- [ ] **Step 1: Write failing tests**
+
+Use existing MP4 test helpers to build:
+
+- one init segment containing `ftyp+moov`
+- two `moof+mdat` media pairs on path `/ingest/video`
+- two `moof+mdat` media pairs on path `/ingest/audio`
+
+Assert:
+
+- no media object is emitted before init is available
+- a catalog object is emitted first
+- media objects retain path-specific track names
+- path queues can interleave without blocking each other
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run:
+
+```bash
+cmake --build build --target openmoq-publisher-live-dash-tests
+./build/openmoq-publisher-live-dash-tests
+```
+
+Expected: tests fail because the adapter does not exist.
+
+- [ ] **Step 3: Implement adapter**
+
+Implement a queue-backed `LiveDashIngestSession`:
+
+- `ingest(path, bytes)` appends bytes to that path's `StreamingMp4Reader`.
+- `ftyp` and `moov` are buffered as init bytes.
+- tracks are extracted after `moov`.
+- `moof` is held until the following `mdat`.
+- each complete `moof+mdat` becomes a `LiveObject`.
+- group IDs increment independently per track/path.
+- `source()` returns a `LiveObjectSource` with all known tracks plus `catalog`.
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run the same test command. Expected: adapter tests pass.
+
+## Task 4: HTTP Server and Multiple Concurrent Paths
+
+**Files:**
+- Modify: `src/live_dash_ingest.cpp`
+- Test: `tests/live_dash_ingest_test.cpp`
+
+- [ ] **Step 1: Write failing tests**
+
+Start the ingest server on loopback port `0`, open two client sockets, and send chunked `PUT` requests to `/ingest/video` and `/ingest/audio`. Assert both paths produce objects and that a malformed request gets a `400` response without killing the listener.
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run:
+
+```bash
+cmake --build build --target openmoq-publisher-live-dash-tests
+./build/openmoq-publisher-live-dash-tests
+```
+
+Expected: tests fail because server handling is not implemented.
+
+- [ ] **Step 3: Implement server**
+
+Implement `LiveDashIngestServer`:
+
+- bind/listen on configured host/port
+- expose actual bound port for tests
+- accept each connection in its own worker thread
+- parse request line and headers
+- allow only `POST` and `PUT`
+- require path prefix match
+- decode `Transfer-Encoding: chunked`
+- feed decoded bytes into the shared session
+- send `204 No Content` on clean completion and `400`/`405` on invalid input
+- stop cleanly when requested
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run the same test command. Expected: HTTP server tests pass.
+
+## Task 5: CLI and Publisher Wiring
+
+**Files:**
+- Modify: `include/openmoq/publisher/cli_options.h`
+- Modify: `src/cli_options.cpp`
+- Modify: `src/main.cpp`
+- Test: `tests/cli_options_test.cpp`
+
+- [ ] **Step 1: Write failing CLI tests**
+
+Assert:
+
+```cpp
+--live-source dash --dash-listen 127.0.0.1:8080 --dash-path /ingest --endpoint https://relay.example.com:443/moq
+```
+
+parses successfully, while missing `--dash-listen` or missing `--endpoint` fails.
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run:
+
+```bash
+cmake --build build --target openmoq-publisher-cli-tests
+./build/openmoq-publisher-cli-tests
+```
+
+Expected: tests fail because the CLI mode is not implemented.
+
+- [ ] **Step 3: Implement CLI and main wiring**
+
+Add:
+
+- `LiveSourceKind::kDash`
+- `--dash-listen <host:port>`
+- `--dash-path <prefix>` defaulting to `/ingest`
+- `--dash-queue-depth <count>` defaulting to `128`
+
+In `main.cpp`, start `LiveDashIngestServer`, then publish `server.source()` with `Publisher::publish_live_objects(...)`.
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run the CLI tests. Expected: all CLI tests pass.
+
+## Task 6: Docs and Final Verification
+
+**Files:**
+- Modify: `README.md`
+- Modify: `docs/quickstart.md`
+
+- [ ] **Step 1: Document usage**
+
+Add a short CTE LL-DASH section with the publisher command and a chunked HTTP producer example.
+
+- [ ] **Step 2: Run complete verification**
+
+Run:
+
+```bash
+cmake --build build
+ctest --test-dir build --output-on-failure
+git diff --check
+```
+
+Expected: build succeeds, tests pass, and whitespace check passes.
+
+- [ ] **Step 3: Review C++ changes**
+
+Review `git diff HEAD` for correctness, concurrency, ownership, parser overflow handling, and test coverage before commit.
+