livekit
Background segmentation flickering too much
On our app we are using both the BackgroundBlur(50) and VirtualBackground(imagePath) processors, and for a large amount of users there are some areas of the video that constantly flicker (switch between being recognized as a background or not on almost every frame) that are distracting, increase anxiety and leads to low perceived quality.
Here it is a short video of it happening to me, but this issue is worse for most of our users that have curly hair, larger chair on the back or more objects on their background. https://github.com/user-attachments/assets/8a45d196-85df-4e13-a403-6182e95da900
The specific issue isn't that the segmentation is not good enough, but that on every frame it flips between considering some areas as a background or not. Even when users are standing still and there is almost no change on the image frame.
I tried looking for params or ways to customize this, but I didn't find any that could improve this behavior.
Have same issue, I think it might because of the OpenGL script that apply blur on segmented video frame. It might not optimize or not handle the border of segment really well or not at all handled.
I read this nice article of how google implement the blur effect and it seem that they use joint bilateral filter to handle that https://research.google/blog/background-features-in-google-meet-powered-by-web-ml/
for segmentation this plugin uses https://github.com/google-ai-edge/mediapipe directly.
There are limited options for us to influence segmentation and thus the flickering of the segmentation mask. If you find any options to tweak the mediapipe segmentation for a better perceived user experience, we'd be happy to add support for those tweaks
Even if this feature isn’t the most critical in my opinion, some of my users still experience difficulty using our videoconference product because of the quality of the background blur. For remote workers, not being confident that their home environment is reliably hidden can be a significant source of stress.
If anyone is interested in working on this topic, please feel free to contact me. I’m considering retraining a model to improve blur quality, especially around challenging areas like hair, edges, motion, and varied lighting conditions. Approaches like the high-definition segmentation used in Google Meet [1] show that significant improvements are possible by combining advanced deep learning techniques with real-time optimization. The goal is to make users feel secure and confident while using our platform, even in less-than-perfect home setups.
[1] High-Definition Segmentation in Google Meet: https://research.google/blog/high-definition-segmentation-in-google-meet/
Hello, we face the same issues. How can we become in contact. We are also thinking abour retraining the modell because google meets tflite is not opensource at the moment.
for segmentation this plugin uses https://github.com/google-ai-edge/mediapipe directly.
There are limited options for us to influence segmentation and thus the flickering of the segmentation mask. If you find any options to tweak the mediapipe segmentation for a better perceived user experience, we'd be happy to add support for those tweaks
Hello @lukasIO I face the issue with the flickering. We use the latest version of the project. But our mask completly divers to the mediapippe mask. The mask in the mediapipe sample is much "quiter" and not so "fast" in flickering.