Background blurring 👤✨

[rapterjet2004]

• fixes Allow background blur during video calls #5349

This PR implements Background Blurring, an optional (by default off) effect that, like web and iOS, uses a low-memory on-device AI model to create a mask between the foreground/background, before using openCV to perform gaussian blur, along with several other file format conversions, necessary for image processing Y'UV frames to RGBA and vice-versa. In the future this PR also makes it easier to implement additional background effects, like background images, background videos/gifs, party hats, company logo overlays, and more.

Adds new files

• BackgroundBlurFrameProcessor an implementation of VideoProcessor which is an interface that WebRTC provides to perform operations on camera frames, and ImageSegmentationHelper.SegmenterListener, which is a small interface I wrote to handle asynchronous passing of camera frames from MediaPipe (You can actually handle it synchronously too, since the VideoProcessor operates on it's own thread, but this is cleaner)
• ImageSegmentationHelper an edited version of google's. Basically a wrapper over MediaPipe's ImageSegmenter which uses the on device AI Model to segment images between the foreground and background.
• BlurBackgroundViewModel handles the state of the blur background functionality, which is by default off
• download_model.gradle a small gradle file to download the on device AI Model on compile time, if not already found. The model is pretty light, just a couple hundred KBs. Also it's licensed for Commercial use under Apache v2.
• background_replace_24px - Background Blur icon

🖼️ Screenshots

Mask	Post Process

🚧 TODO

• Fix bug where turning off background blur stops frames from being sent over WebRTC. ( it was a timestamp issue )
• Optimize object creation, passing, and processing ( still opportunities for optimization, but trivial time savings )
• Research GPU acceleration ( not maintainable, openGL ES has too high learning curve for future devs )
• Bug on orientation change, the camera is flipped upside down, change the flip code on detection of landscape in onFrameCaptured
• The blur background processor crops the Video Frame to square, look into preserving aspect ratio ( Not possible the Selfie Segmentation Model only outputs square or landscape )

🏁 Checklist

• ⛑️ Tests (unit and/or integration) are included or not needed
• 🔖 Capability is checked or not needed
• 🔙 Backport requests are created or not needed: /backport to stable-xx.x
• 📅 Milestone is set
• 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

[sowjanyakch]

Tested this PR.

1. When background blur is enabled, the front camera is slightly zoomed compared to when blur is off.
2. For audio-only calls, the background blur button should be hidden to avoid unnecessary resource usage. Wdyt?
3. If you answer a video call, turn on background blur, and then stop your camera and then turn on camera, the person on the other end sees a blank screen. The video is displayed only a certain delay. (here delay is noticeable).
4. If you answer a video call, turn off background blur, and then stop your camera and then turn on camera, the person on the other end sees a blank screen. The video is displayed only a certain delay. (here delay is small).

[sowjanyakch]

see comment #5387 (comment)

[rapterjet2004]

1. When background blur is enabled, the front camera is slightly zoomed compared to when blur is off.

Not possible to fix, as the Selfie Segmentation Model only outputs square or landscape. Besides that, I removed the option for audio only calls, and made sure to turn off background blur when the camera is off, to avoid errors with state.

[github-actions]

APK file: https://www.kaminsky.me/nc-dev/android-artifacts/5387.apk



To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.

[github-actions]

Codacy

Lint

Type	master	PR
Warnings	99	99
Errors	0	0

SpotBugs

Category	Base	New
Bad practice	6	6
Correctness	11	11
Dodgy code	62	62
Internationalization	3	3
Malicious code vulnerability	3	3
Performance	4	4
Security	1	1
Total	90	90

[sowjanyakch]

If you answer a video call, turn on background blur, and then stop your camera and then turn on camera, the person on the other end sees a blank screen. The video is displayed only a certain delay. (here delay is noticeable).

the above problem still exists for me.