[next] [6] Add integration test suite with dynamic ports and parallel execution by joaoantoniocardoso · Pull Request #594 · mavlink/mavlink-camera-manager

joaoantoniocardoso · 2026-03-26T05:30:57Z

Summary

Adds the clients crate, a comprehensive integration test suite with dynamic port allocation, a latency comparison example, and CI updates for nextest plus multi-GStreamer compatibility coverage.

Review Progression

The incremental part of this PR is organized into 22 commits across 6 phases.

Phase 1: Existing source modifications (commits 1-2)

Two small source changes that the new client/test code depends on. Commit 1 adds Deserialize derives and a to_nanos() helper to zenoh_message types so the Zenoh client can deserialize frames. Commit 2 removes the old WebRTC leak-detection helper and its now-unused CLI hook because this PR replaces that path with broader browser/integration coverage.

Phase 2: Cargo manifests (commits 3-5)

Cargo-only changes. Adds the new clients workspace member manifest, registers it in the workspace root together with test/example entries, then updates the lock file.

Phase 3: `clients` crate (commits 6-13)

The new library crate, starting with the shared abstractions (StreamClient, Codec, frame probe helpers) and signalling protocol types, followed by one commit per client implementation: thumbnail, RTSP, UDP, Zenoh, and WebRTC. The final commit in this phase reorders module declarations.

Phase 4: Integration test suite (commits 14-19)

The harness comes first: McmProcess with ephemeral ports, a REST client wrapper, and polling/state helpers. Then the test modules build outward in scope:

data_flow: end-to-end verification for every sink type
lazy_lifecycle: state transitions, idle timing, recovery, and per-sink lifecycle checks
mavlink_heartbeat: heartbeat persistence through idle cycles
redirect_webrtc: redirect pipelines combined with WebRTC/thumbnail/Zenoh consumers
remote_integration: heavier fake/real camera scenarios, mixed concurrent consumers, and the disabled-ONVIF coverage that required small source-side guard rails

Phase 5: Example (commit 20)

A latency measurement CLI tool that receives the same stream via RTSP, WebRTC, and UDP simultaneously, correlates frames by content hash, and computes pairwise latency deltas. Includes a companion Python plotting script.

Phase 6: CI configuration (commits 21-22)

Moves CI from cargo test to cargo nextest run, adds gstreamer1.0-plugins-bad, introduces .config/nextest.toml, and adds the multi-GStreamer compatibility workflow.

Test plan

cargo nextest run
Tests can run in parallel without port conflicts
Fake-mode tests work without external hardware
Disabled-ONVIF scenarios are covered
CI includes multi-GStreamer compatibility coverage

…on advance

… and to_nanos helper Add `Deserialize` derive to `Timestamp` and `CompressedVideo` so the new `stream-clients` crate (added in a later commit) can deserialize Zenoh messages received over the wire. The zenoh_client includes this file via `#[path]` and needs both derives for CDR deserialization. Add `Timestamp::to_nanos()` to convert a timestamp to nanoseconds, used by the stream-latency measurement example. Add `#[allow(dead_code)]` on `impl Timestamp` since some methods are only used when the file is included from the clients crate.

Removes the in-process WebRTC thread leak test that was gated behind the webrtc-test cargo feature: - Deletes src/lib/helper/develop.rs - Removes --enable-webrtc-task-test CLI argument from manager.rs - Removes webrtc-test feature gate from helper/mod.rs and main.rs Thread leak detection is now a standard integration test (thread_leak::test_webrtc_thread_leak) that inspects the MCM child process externally via sysinfo, requiring no special feature flags or CLI arguments.

New workspace member `stream-clients` (at clients/) providing GStreamer-based stream consumer clients for integration testing and latency measurement: - RTSP, UDP, WebRTC, Zenoh, and thumbnail HTTP clients - Optional `webrtc-test` feature gates the Selenium-based browser_webrtc_client (depends on thirtyfour) Dependencies include gstreamer 0.25, async-tungstenite, zenoh 1.5.1, reqwest, and serde/serde_json for signalling protocol handling.

…ncies Register the new `stream-clients` crate (clients/) as a workspace member. Set `autobenches = false` and `autotests = false` to disable automatic test/bench discovery, since the integration tests use an explicit `[[test]]` entry pointing at `tests/integration/main.rs`. Extend the `webrtc-test` feature to forward `stream-clients/webrtc-test` so the Selenium-based browser WebRTC client is available when the feature is enabled. Add dev-dependencies required by the integration test suite: reqwest, tokio-tungstenite, anyhow, futures, tempfile, thirtyfour, zenoh, stream-clients, and gst-rtsp-server. Add `[[example]]` entry for the `stream_latency` measurement tool.

Core abstractions for the stream-clients crate, shared by all stream consumer clients. Public API: - `StreamClient` async trait with `frames()`, `pipeline()`, `wait_for_frames(min, timeout)`, and `wait_for_continuous_frames(duration, check_interval)` for frame-count polling with timeout and stall detection. - `Codec` enum: H264, H265, Mjpg, Yuyv, Rgb. - `FrameSample` struct: content_hash, relative_pts_ms, arrival instant, and buffer_size for per-frame metadata. - `SampleSender` type alias for `mpsc::UnboundedSender<FrameSample>`. - `hash_vcl_nals(data)`: hashes only H.264/H.265 VCL NAL units, producing a stable hash across different pipeline processing chains (SPS/PPS injection, stream-format conversion, etc.). - `attach_frame_probe(pad, name, sender)`: installs a GStreamer pad probe that sends a `FrameSample` for each buffer, enabling hash-based frame correlation across transports.

Mirror of the main application's WebRTC signalling types, without the ts-rs dependency. Used by `webrtc_client` and integration tests to communicate over the WebSocket signalling channel. Defines the JSON message envelope (`Message`, `Question`, `Answer`, `Negotiation`), session/peer ID types (`SessionId`, `PeerId` as UUID aliases), stream metadata (`Stream`, `MediaNegotiation`), and SDP/ICE exchange types (`RTCSessionDescription`, `RTCIceCandidateInit`, `BindOffer`, `BindAnswer`). Includes `From` impls to wrap each variant into the `Protocol` enum for ergonomic message construction.

HTTP client for MCM's `/thumbnail?source=` endpoint, used by integration tests to verify thumbnail generation. Public API: - `ThumbnailClient::new(base_url)`: creates a client targeting the given MCM REST base URL. - `thumbnail(source)`: single GET request, returns the raw JPEG bytes or an error if the response is not 200. - `thumbnail_with_retry(source, retries, delay)`: retries up to N times with a fixed delay between attempts. - `cold_thumbnail(source, timeout)`: polls until a non-empty thumbnail is returned, used for cold-start scenarios where the pipeline may not yet have produced a frame. - `ensure_data_flowing(source, count, interval)`: fetches `count` thumbnails at the given interval, asserting each is non-empty, to verify continuous data flow.

GStreamer pipeline client that pulls from an RTSP URL, depayloads by codec (H264/H265/MJPEG), parses where applicable, and ends in fakesink. Waits for TCP reachability on the RTSP host:port before building the pipeline, avoiding spurious failures when MCM's RTSP server is still starting up. Optionally accepts a `SampleSender` to install a frame probe via `attach_frame_probe` for hash-based frame correlation in tests. Public API: - `RtspClient::new(url, codec, sender)`: builds and starts the pipeline after confirming TCP reachability. - Implements `StreamClient` (frames(), pipeline()). - `Drop` sets the pipeline to Null.

GStreamer udpsrc-based client for receiving RTP video streams over UDP. Supports two construction modes: - `new(port, codec, sender)`: for coded streams (H264/H265/MJPEG), builds the appropriate RTP depayloader and parser chain. - `new_raw(port, codec, width, height, sender)`: for raw streams (YUYV/RGB) with explicit dimensions passed via capsfilter. Both modes end in fakesink with frame probe attached via `attach_frame_probe` for hash-based flow verification in tests. Implements `StreamClient` (frames(), pipeline()). Drop sets the pipeline to Null.

Subscribes to a Zenoh topic and deserializes CompressedVideo messages (CDR-encoded) into GStreamer buffers for frame counting and hashing. The CompressedVideo type is imported via `#[path]` from the main crate's `zenoh_message.rs` (commit 1 added the required Deserialize derives). Pipeline: appsrc -> h264parse/h265parse -> capsfilter -> fakesink, with frame probe attached for hash-based flow verification. Supports H264 and H265 codecs. The Zenoh subscriber runs in a background tokio task that pushes buffers into appsrc. Public API: - `ZenohClient::new(topic, codec, sender, zenoh_config)`: opens a Zenoh session, subscribes to the topic, and starts the pipeline. - Implements `StreamClient` (frames(), pipeline()). - `Drop` sets the pipeline to Null and aborts the subscriber task.

Full GStreamer webrtcbin-based consumer client that handles the complete WebRTC session lifecycle over WebSocket signalling. Signalling flow: 1. Connect to the MCM signalling WebSocket endpoint 2. Obtain a peer ID 3. List available streams and select one (optionally by producer ID) 4. Start a session and receive the SDP offer 5. Create an SDP answer via webrtcbin and send it back 6. Exchange ICE candidates bidirectionally Media handling: - Dynamic pad callback detects codec from caps (H264/H265) - Builds depay -> parse -> decode pipeline per pad - Frame counting on both parsed (pre-decode) and decoded paths - Optional SampleSender probe on the parsed path for hash-based frame correlation Public API: - `WebrtcClient::connect(ws_url, producer, codec, sender)`: async constructor that completes the signalling handshake and returns a connected client. - `decoded_frames()`: frame count from the decoded path. - `wait_for_decoded_frames(min, timeout)`: polls decoded frame count. - Implements `StreamClient` (frames(), pipeline()) on the parsed path. - `Drop` aborts the signalling task and sets pipeline to Null.

Moves thumbnail_client below the other protocol clients for consistent alphabetical ordering in lib.rs.

Introduce the integration test binary at tests/integration/main.rs and the shared common module used by all test categories. The common module provides three submodules: - types: Serde models mirroring MCM's REST and signalling API (StreamInformation, PostStream, StreamStatus, StreamStatusState, ExtendedConfiguration, SignallingProtocol/Message/Question/Answer, BindOffer, BindAnswer, etc.). These allow tests to create streams, check status, and communicate over WebSocket without depending on the main crate's types. - mcm: McmProcess helper that spawns the mavlink-camera-manager binary as a child process with ephemeral ports (REST, signalling, RTSP) and a temporary settings directory. Includes automatic port-conflict retry, allocate_ports() and allocate_udp_ports() helpers, and optional MAVLink/Zenoh endpoint configuration. - api: McmClient HTTP client wrapping reqwest for REST API operations (list/create/delete streams, start/end WebRTC sessions, get thumbnails, check extended configuration). Also provides StateMonitor which polls stream status in a background task and records state transitions for assertions in lifecycle tests.

End-to-end data flow verification for each sink type. Tests create a stream via the REST API, connect a stream-clients consumer, and verify frames arrive continuously over a measurement window. mod.rs provides shared helpers: - NON_LAZY ExtendedConfiguration (disables lazy lifecycle for deterministic testing) - drain/collect_frames/wait_first_frame for frame sample analysis - verify_data_flow: asserts continuous frame arrival over the measurement window with gap detection - RTSP and TCP readiness polling helpers - create_fake_rtsp / create_redirect_stream setup functions Test submodules: - rtsp: H264 fake RTSP stream via RTSP client - udp: H264 fake RTSP stream via UDP sink client - webrtc: H264 fake RTSP stream via WebRTC client - thumbnail: thumbnail endpoint data flow check - qr: QR-code overlay verification (encodes stream name in video, decodes from received frames to verify correct routing) - redirect: redirect pipeline tests for H264, H265, and MJPEG with both UDP and WebRTC consumers - zenoh: Zenoh topic subscription tests for H264 and H265, including Zenoh-specific configuration and subscriber setup

Tests for the lazy stream lifecycle state machine, verifying that streams correctly transition through Idle, Running, Draining, and back to Idle based on consumer activity. mod.rs provides shared helpers: - IDLE_GRACE (5s) and IDLE_WAIT (8s) constants matching the watcher's idle grace period plus slack for the 100ms tick loop - setup_fake_rtsp: creates an MCM instance with a fake RTSP stream - wait_for_idle: polls until StreamStatusState::Idle is observed - rtsp_options_ok: RTSP OPTIONS probe to check server readiness Test submodules: - state_transitions: verifies the full Idle -> Running -> Draining -> Idle cycle using StateMonitor transition recording - rtsp: RTSP consumer triggers Running, disconnection triggers Idle - webrtc: WebRTC session start/end drives lifecycle transitions - thumbnail: thumbnail requests keep the stream alive - recovery: stream recovers from Idle when a new consumer connects - mixed: concurrent RTSP + WebRTC consumers, stream stays Running until all consumers disconnect

…nce test Verifies that MAVLink HEARTBEAT messages with MAV_TYPE_CAMERA continue to be emitted even after a lazy stream transitions to Idle. Test procedure: 1. Spawn MCM with a MAVLink UDP endpoint (udpout:127.0.0.1:<port>) 2. Create a fake RTSP stream with MAVLink enabled and lazy lifecycle 3. Wait for the stream to go through Running -> Draining -> Idle 4. Listen for heartbeats over ~5 seconds on the UDP port 5. Assert at least 3 heartbeats were received (1 Hz expected rate) Uses a background std::thread with the mavlink crate for UDP reception, since the mavlink crate's blocking recv is simpler than async alternatives for this use case.

…egration tests Tests for redirect pipelines (where an external source sends RTP packets to MCM) combined with WebRTC, thumbnail, and Zenoh consumers. mod.rs provides: - spawn_h264_udp_sender: launches an external gst-launch-1.0 process that sends H264 RTP to a given host:port, simulating a real camera or external encoder - Shared constants and imports for the submodules Test submodules: - webrtc: verifies that a redirect stream (UDP H264 input) can be consumed via WebRTC signalling, testing both the producer-specific session start and frame reception - thumbnail: verifies thumbnail generation from redirect streams - zenoh: verifies Zenoh topic publication from redirect streams, including subscriber setup and frame reception for H264

… tests Heavier integration scenarios supporting both fake and real camera sources. Designed for CI with fake sources and manual runs against real cameras via the TEST_SOURCE_MODE=camera environment variable. mod.rs provides: - TestEnv: auto-selects between spawning a local MCM with fake RTSP source or connecting to a pre-running instance (camera mode reads MCM_REST_URL, MCM_SIGNALLING_PORT, MCM_RTSP_URL, STREAM_SOURCE, ICE_FILTER_REGEX env vars) - mcm_pid() for external thread inspection via sysinfo - SourceMode/SourceTag enums for test filtering - Shared setup helpers (wait_for_stream_running, verify_frames, create_stream_and_wait) that handle both fake and camera modes - STATE_POLL and SETUP_TIMEOUT constants Test submodules: - rtsp: RTSP consumption with frame counting and continuous flow - webrtc: WebRTC session lifecycle with frame verification - thumb: thumbnail endpoint verification with cold-start and continuous data flow checks - zenoh: Zenoh topic subscription (camera mode only, skipped in fake) - mixed: concurrent multi-consumer scenarios (RTSP + WebRTC + thumbnail simultaneously) - thread_leak: WebRTC thread leak detection using sysinfo to monitor the MCM process for leaked threads after repeated WebRTC session cycles. Uses a deny-list of known infrastructure thread prefixes and a warmup cycle with dynamic stabilization for robust baselines.

CLI tool that receives the same video stream via multiple transports (RTSP, WebRTC, UDP) simultaneously and measures pairwise latency differences by correlating frames across transports using VCL NAL content hashing. Usage: cargo run --example stream_latency -- \ --rtsp rtsp://host/path --webrtc ws://host:port [--udp 0.0.0.0:5600] Features: - Configurable warmup period to exclude startup transients - Periodic progress reports during measurement - CSV export (--csv output.csv) of raw per-frame data for post-processing - Per-transport statistics: frame count, bitrate, inter-frame jitter - Pairwise latency: mean, median, stddev, min, max for each transport pair that shares matching frames Includes a companion Python script (plot_results.py) that loads one or more CSV files and plots latency comparisons across runs (e.g. different bitrates) and across transport pairs using matplotlib.

Switch CI from cargo test to cargo nextest run for parallel test execution with per-test timeout enforcement. Changes to .github/workflows/rust.yml: - Add gstreamer1.0-plugins-bad to apt dependencies (required by some integration test pipelines that use elements from plugins-bad) - Install nextest via taiki-e/install-action@nextest - Replace `cargo test --verbose --locked` with `cargo nextest run --verbose --no-fail-fast --locked` - Set GST_DEBUG=2 and GST_DEBUG_NO_COLOR=1 env vars for GStreamer traceability in CI logs - Add --success-output final --failure-output final for cleaner nextest output Add .config/nextest.toml with: - 120s slow-timeout with terminate-after=2 (kills hung tests) - A `camera` profile with threads-required=1 override for remote_integration:: tests, which share a single MCM instance on the remote host and must run sequentially (usage: cargo nextest run --profile camera)

patrickelectric · 2026-05-25T18:55:20Z

The CI is failing hard in this PR

joaoantoniocardoso · 2026-05-25T22:39:37Z

The CI is failing hard in this PR

Yes, intentionally. It will only pass once we merge some incoming fixes (and I rebase it over master).

Please review the design regardless of the current CI results.

joaoantoniocardoso mentioned this pull request Mar 26, 2026

[next] [5] Rework stream architecture with lifecycle state machine and latency optimizations #593

Merged

7 tasks

joaoantoniocardoso force-pushed the pr/integration-test-suite branch 6 times, most recently from 6090be5 to 9a6cdeb Compare March 26, 2026 17:25

joaoantoniocardoso changed the title ~~[next] Add integration test suite with dynamic ports and parallel execution~~ [next] [6] Add integration test suite with dynamic ports and parallel execution Mar 26, 2026

joaoantoniocardoso force-pushed the pr/integration-test-suite branch from 9a6cdeb to 35bd6f2 Compare March 27, 2026 02:32

joaoantoniocardoso mentioned this pull request Mar 28, 2026

[next] Add stream latency measurement tool #592

Closed

3 tasks

joaoantoniocardoso force-pushed the pr/integration-test-suite branch 4 times, most recently from 679b61d to c68b7cc Compare April 3, 2026 00:02

joaoantoniocardoso force-pushed the pr/integration-test-suite branch 10 times, most recently from 787a1fc to 7018cf4 Compare April 8, 2026 23:48

joaoantoniocardoso commented Apr 15, 2026

View reviewed changes

Comment thread Cargo.toml Outdated

joaoantoniocardoso force-pushed the pr/integration-test-suite branch 3 times, most recently from ea05b97 to 0540cbf Compare April 16, 2026 21:25

joaoantoniocardoso mentioned this pull request Apr 17, 2026

Add file sink #601

Draft

4 tasks

joaoantoniocardoso marked this pull request as ready for review April 17, 2026 17:35

joaoantoniocardoso added 27 commits May 22, 2026 13:49

src: lib: stream: Adopt actor-based lifecycle and remove legacy module

a4c657e

src: lib: stream: Tear down RTSP factories on Stream drop

a8bf762

src: lib: stream: Distinguish backoff in stream status

76410f5

src: lib: stream: manager: Use post-add snapshot for thumbnail positi…

48408f5

…on advance

cargo: Update lock file

a34ea76

clients: lib: Reorder module declarations

89d036b

Moves thumbnail_client below the other protocol clients for consistent alphabetical ordering in lib.rs.

tests: integration: redirect_webrtc: Add WebRTC profile tests

0be0486

.github: workflows: test_compatibility: Add multi-GStreamer version CI

e9533e3

joaoantoniocardoso force-pushed the pr/integration-test-suite branch from d74ce66 to e9533e3 Compare May 22, 2026 18:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[next] [6] Add integration test suite with dynamic ports and parallel execution#594

[next] [6] Add integration test suite with dynamic ports and parallel execution#594
joaoantoniocardoso wants to merge 29 commits into
mavlink:masterfrom
joaoantoniocardoso:pr/integration-test-suite

joaoantoniocardoso commented Mar 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

patrickelectric commented May 25, 2026

Uh oh!

joaoantoniocardoso commented May 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joaoantoniocardoso commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review Progression

Phase 1: Existing source modifications (commits 1-2)

Phase 2: Cargo manifests (commits 3-5)

Phase 3: clients crate (commits 6-13)

Phase 4: Integration test suite (commits 14-19)

Phase 5: Example (commit 20)

Phase 6: CI configuration (commits 21-22)

Test plan

Uh oh!

Uh oh!

patrickelectric commented May 25, 2026

Uh oh!

joaoantoniocardoso commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joaoantoniocardoso commented Mar 26, 2026 •

edited

Loading

Phase 3: `clients` crate (commits 6-13)

joaoantoniocardoso commented May 25, 2026 •

edited

Loading