obsproject
linux-pipewire: Add PipeWire audio captures
Description
Adds three audio capture sources that utilize PipeWire to capture audio inputs, outputs, as well as audio of specific applications.
-
pipewire-audio.h/pipewire-audio.c: These contain some basic wrappers over PipeWire stuff in order to avoid duplicated code. This is where you will find the PipeWire instance creation/destruction and audio output code. -
pipewire-audio-capture-device.c: The audio input/output device capture source. Uses PipeWire streams to get audio from sink and source nodes in the PipeWire graph. -
pipewire-audio-capture-app.c: The app audio capture source. Uses a virtual sink to which all targeted app audio output streams are connected in order for them to be mixed by PipeWire. Then uses a PipeWire stream that gets connected to that virtual sink. Another approach would be to use custom PipeWire nodes but this would add too much code and room for error (the virtual sink used is made up of 1k lines in PipeWire).
Motivation and Context
PipeWire is increasingly getting adopted by users as well as major distros (like Fedora), and is on track to be the de facto standard for audio on Linux and others. The PipeWire project has created a PulseAudio-PipeWire bridge, so the current PulseAudio sources work as intended. However, pipewire-pulse has had its issues in the past, although they were minor. Compared to the current PulseAudio source, audio has slightly lower latency. Also, this automatically updates when a targeted device is reconnected or its channels are changed. Additionally, PipeWire allows us to capture specific application audio, something that currently needs external tools and tinkering to get working.
How Has This Been Tested?
This has been tested on Arch Linux, on KDE Plasma (X11 and Wayland) and on latest Fedora Gnome on Wayland by
- Using Helvum to keep track of the PipeWire graph
- Spinning up to 20 Firefox tabs playing audio along with other random apps and having many sources with different possible configurations
- Randomly closing/opening targets the captures are streaming from
- Re-plugging devices the captures are streaming from
- Switching default devices and changing device channels from KDE Plasma's audio settings
Captures kept on going, no errors reported by PipeWire
Types of changes
- New feature (non-breaking change which adds functionality)
Checklist:
- [x] My code has been run through clang-format.
- [x] I have read the contributing document.
- [x] My code is not on the master branch.
- [x] The code has been tested.
- [x] All commit messages are properly formatted and commits squashed where appropriate.
- [x] I have included updates to all appropriate documentation.
Note: Running OBS with OBS_USE_EGL=1 allows PipeWire capture being loaded under X11.
what occurs on audio underruns or overruns with this code, i dont see any callbacks relating (but maybe i missed it).
what occurs on audio underruns or overruns with this code, i dont see any callbacks relating (but maybe i missed it).
(This is my first time working with audio, please excuse me if I don't understand something quickly)
No, I haven't done anything related to this problem. Sometimes when X and Nvidia were being stupid and my system lagged a bit the audio from the sources was crackling (but the sound from my speakers was fine), was that an underrun?
Also I don't know if it matters but it only provides OBS with data when it has data, it doesn't poll the stream for data.
How do other plugins mitigate this problem?
How do other plugins mitigate this problem?
I believe alsa and sndio just drop out in the case of underruns. The pulseaudio plugin attempts to increase latency and grow the audio buffer, but i would caution against reading its code for correctness as its probably not correctly implemented.
Sometimes when X and Nvidia were being stupid and my system lagged a bit the audio from the sources was crackling (but the sound from my speakers was fine), was that an underrun?
Yes that is the typical result of dropping audio during underruns.
I should say that windows also will increase latency to grow the audio buffer (and is implemented correctly). This is our preferred approach as audio dropping (static) in recordings is general less desirable than XXXms of latency.
I should say that windows also will increase latency to grow the audio buffer (and is implemented correctly). This is our preferred approach as audio dropping (static) in recordings is general less desirable than XXXms of latency.
So to deal with underruns it should request a higher latency from PipeWire then, is that correct?
Played a little bit around with this plugin and noticed that the application capture will be silent, if the application is not longer connected to a speaker. I tested vlc and mpd (pulse and pipewire backend) and disconnected them from my speakers with carla. At this point obs didn't record sound anymore. Maybe we are missing a PipeWire property?
My current guess is that there is no driver in the loop. mpd has the state playing but it won't progress.
Played a little bit around with this plugin and noticed that the application capture will be silent, if the application is not longer connected to a speaker. I tested vlc and mpd (pulse and pipewire backend) and disconnected them from my speakers with carla. At this point obs didn't record sound anymore. Maybe we are missing a PipeWire property?
My current guess is that there is no driver in the loop. mpd has the state playing but it won't progress.
Just tried that with Helvum, the stream gets set to paused
Maybe we are missing a PipeWire property?
Yep, found it (found one?). PW_KEY_NODE_ALWAYS_PROCESS. Maybe there's a more suitable one...?
Edit: Tried a bunch that could be related, they weren't, it doesn't seem to break anything so I'll be keeping this
PW_KEY_NODE_ALWAYS_PROCESS. Maybe there's a more suitable one...?
Nice, but now the stream won't stop, when we switched scenes (I added show and hide callbacks analog to the screencast). Maybe we can ask at #pipewire on IRC.
But I don't know if the pause on hide is wanted when streaming or not.
PW_KEY_NODE_ALWAYS_PROCESS. Maybe there's a more suitable one...?
Nice, but now the stream won't stop, when we switched scenes (I added show and hide callbacks analog to the screencast). Maybe we can ask at #pipewire on IRC.
But I don't know if the pause on hide is wanted when streaming or not.
I was curious so I looked through the PipeWire source code, seems this flag just sets the want_driver prop, but it could do other stuff too. Maybe try the flag PW_KEY_NODE_WANT_DRIVER instead
Completely reworked the app capture, it now uses a virtual sink to mix all the app streams. It had to be seperated from the device capture. Tried to keep it DRY as much as possible, but more improvements can be made. Also the device capture logic for connecting to default devices has changed, there wasn't any actual logic to begin with, it was just letting PipeWire decide. I've learned not to say when a piece of software will be finished but I want to believe this is nearly finished.
https://github.com/obsproject/obs-studio/pull/6645 did some refactoring to split the portal code from the pipewire code for video streams in case this causes any conflicts as it renames some code. I'm not sure if there is anything that can be shared between the video stream code and this either.
#6645 did some refactoring to split the portal code from the pipewire code for video streams in case this causes any conflicts as it renames some code. I'm not sure if there is anything that can be shared between the video stream code and this either.
The latest changes were rebased on master just before being pushed and that PR is included. Haven't noticed any conflicts or problems yet. I didn't touch any existing PipeWire code because from what it seems it's just for video as you said. I added everything audio related to the pipewire-audio files but I can see how we can put audio stuff in pipewire.c and .h.
I added everything audio related to the pipewire-audio files but I can see how we can put audio stuff in pipewire.c and .h.
My idea with #6645 is that all PipeWire code, even audio code, would be handled in there. Would you be so kind to check if it moving the code from pipewire-audio.* into pipewire.* is going to be too disruptive? It's fair game to increase the API surface of pipewire.h to cover the audio front.
Would you be so kind to check if it moving the code from
pipewire-audio.*intopipewire.*is going to be too disruptive?
Simply moving everything only requires renaming some events. Then there is duplicated code for creating a PipeWire loop, context etc and different callbacks for pw_core to work with different structs, so I believe we may need an abstraction for those along with the ones columbarius mentioned in this comment. Also the prefixes are inconsistent (obs_pw_audio_ for audio vs obs_pipewire_ for video)
How would this affect having both PulseAudio and PipeWire as audio capture options? Would it be hidden by default, and would appear when an option is enabled in the settings? Or does it JUST add the PipeWire audio capture functionality? I believe that this merge should also do the first out of what I mentioned, hiding the PulseAudio capture functionality, so that the menu isn't as cluttered (and as such, not as confusing to read).
Would it be hidden by default, and would appear when an option is enabled in the settings? Or does it JUST add the PipeWire audio capture functionality? I believe that this merge should also do the first out of what I mentioned, hiding the PulseAudio capture functionality
This PR only adds the PipeWire audio capture sources. I feel like PulseAudio vs PipeWire source loading should be done in a separate PR, once this is merged
This PR hasn't had any activity in a while, and the plugin version seems to work fine for me, from my testing. Will this be merged anytime soon?
This PR hasn't had any activity in a while, and the plugin version seems to work fine for me, from my testing. Will this be merged anytime soon?
I assume this is because the author had no reason to update it. As its still in draft i wouldn't expect it to be reviewed for merge until the author marks it as ready. We are also in the middle of releasing v28 so its very unlikely anyone will review for a while even if it were undrafted.
@dimtpap Might you take a look at #6669 and tell me if you would just reuse the core/obs_pw api, or if you would include the audio stuff into the stream api?
I was told by Columbarius that work on the unified wrappers has stalled, and since this is ready for general use I can go ahead and mark this as ready
Would love to see this included in OBS. adding this manually is a pain on a Steam Deck.
Is this still planning on being added?
This is still planned, but there is a lot of moving parts. You can follow the discussion and next steps here: https://github.com/obsproject/obs-studio/discussions/7998
Is this still planning on being added?
This is still planned, but there is a lot of moving parts. You can follow the discussion and next steps here: #7998
Thanks for the reply! Will do
Hi, how is this PR going? I see that not much is moving and the linked issues are either all resolved or waiting for approval. This is a great addition to OBS and I think it should be merged as soon as ready. Taking the opportunity to thank everyone involved in the project too!
Hi, how is this PR going? I see that not much is moving and the linked issues are either all resolved or waiting for approval.
Feature wise it's complete. There are ongoing efforts to refactor OBS's PipeWire code to keep it cleaner but are not done. This PR will be refactored to adapt to those changes.