Commit 3413816
committed
feat: Add V3 background segmentation video frame processor
Implement a new CDN-based background segmentation processor that supports
blur (low/medium/high), image replacement, and color replacement effects
with two model types (selfie_general and selfie_multiclass).
Core:
- BackgroundSegmentationVideoFrameProcessor with CDN asset loading
- BackgroundSegmentationAssetLoader with request deduplication
- BackgroundSegmentationCompatibilityChecker (WebGL2, WASM, Workers)
- BackgroundSegmentationMetrics observer for StatsCollector integration
- Dynamic filter switching via setConfig/setModelType without recreation
Observability:
- Video processor metrics (frame rate, per-processor latency) via StatsCollector
- Event publishing for filter lifecycle (backgroundFilterStarted, backgroundFilterConfigSelected, backgroundFilterFailed)
- Extended VideoFXEventAttributes with V3-specific fields
Demo:
- Replace legacy MediaPipe/TensorFlow-based filters with V3 segmentation
- Add blur strength options (low/medium/high) for both model types
- Fix iOS crash on rapid filter switching (concurrency guard)
- Fix filter re-application after offline/online transition
- Remove @tensorflow-models/body-segmentation dependency
Testing:
- Add unit tests for BackgroundSegmentation (processor, asset loader, compatibility checker, metrics)
- Add barrel export coverage for named re-exports from index.ts
- Export BackgroundSegmentationCompatibilityResult, BackgroundSegmentationMetricsObserver, BackgroundSegmentationMetricReport from index.ts for documentation
- Add BackgroundSegmentationTest with blue pixel verification
- Add fake video stream and retry helpers for filter validation1 parent d8c0096 commit 3413816
33 files changed
Lines changed: 2864 additions & 1033 deletions
File tree
- demos/browser
- app/meetingV2
- video/filters
- integration
- configs
- fake_stream
- pages
- tests
- steps
- utils
- script
- src
- audiovideocontroller
- backgroundsegmentation
- devicecontroller
- eventcontroller
- mediastreambroker
- statscollector
- videoframeprocessor
- test
- backgroundsegmentation
- statscollector
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
8 | 31 | | |
9 | 32 | | |
10 | 33 | | |
| |||
0 commit comments