Skip to content

Commit 72f5adb

Browse files
committed
feat: Add V3 background segmentation video frame processor
Implement a new CDN-based background segmentation processor that supports blur (low/medium/high), image replacement, and color replacement effects with two model types (selfie_general and selfie_multiclass). Core: - BackgroundSegmentationVideoFrameProcessor with CDN asset loading - BackgroundSegmentationAssetLoader with request deduplication - BackgroundSegmentationCompatibilityChecker (WebGL2, WASM, Workers) - BackgroundSegmentationMetrics observer for StatsCollector integration - Dynamic filter switching via setConfig/setModelType without recreation Observability: - Video processor metrics (frame rate, per-processor latency) via StatsCollector - Event publishing for filter lifecycle (backgroundFilterStarted, backgroundFilterConfigSelected, backgroundFilterFailed) - Extended VideoFXEventAttributes with V3-specific fields Demo: - Replace legacy MediaPipe/TensorFlow-based filters with V3 segmentation - Add blur strength options (low/medium/high) for both model types - Fix iOS crash on rapid filter switching (concurrency guard) - Fix filter re-application after offline/online transition - Remove @tensorflow-models/body-segmentation dependency Testing: - Add unit tests for BackgroundSegmentation (processor, asset loader, compatibility checker, metrics) - Add barrel export coverage for named re-exports from index.ts - Export BackgroundSegmentationCompatibilityResult, BackgroundSegmentationMetricsObserver, BackgroundSegmentationMetricReport from index.ts for documentation - Update VideoProcessingTest to use V3 background segmentation filters (replaces V2 VideoFx tests) - Add VideoTestSteps utility methods for blur, image replacement, and color replacement with model parameter - Add fake video stream (output.y4m) for consistent pixel-based filter verification - Add blue pixel percentage check and pixel diff verification for filter validation
1 parent d8c0096 commit 72f5adb

32 files changed

Lines changed: 2836 additions & 1065 deletions

CHANGELOG.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,29 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
### Added
11+
12+
- Added `BackgroundSegmentationVideoFrameProcessor` with CDN asset loading, request deduplication, and browser compatibility checking (WebGL2, WASM, Workers)
13+
- Added configurable blur strength (low/medium/high) and support for `selfie_general` and `selfie_multiclass` model types
14+
- Added `BackgroundSegmentationMetrics` observer for StatsCollector integration
15+
- Added dynamic filter switching via `setConfig`/`setModelType` without processor recreation
16+
- Added video processor metrics collection (frame rate, per-processor latency) via StatsCollector
17+
- Added event publishing for filter lifecycle (`backgroundFilterStarted`, `backgroundFilterConfigSelected`, `backgroundFilterFailed`)
18+
- Added V3-specific fields to `VideoFXEventAttributes`
19+
20+
### Removed
21+
22+
- Removed `MediaPipeBodySegmentationProcessor` and `@tensorflow-models/body-segmentation` dependency from demo
23+
24+
### Changed
25+
26+
### Fixed
27+
28+
- Fixed iOS page crash when rapidly switching background filters
29+
- Fixed video filter not re-applying after offline/online transition
30+
831
## [3.32.0] - 2026-05-14
932

1033
### Added

0 commit comments

Comments
 (0)