Add possibility to share screen audio when screensharing is enabled
Goal
This PR adds support for capturing and transmitting screen audio during screen sharing on Android. Previously, when users shared their screen, only the video was transmitted. This change enables capturing system audio (e.g., from media playback, other apps) and mixing it with microphone audio before transmission. Additionally, it ensures that screen audio continues to be transmitted even when the microphone is muted, addressing a common use case where users want to share audio content while remaining muted.
Implementation
Screen Audio Capture:
- Added screen audio capture using
AudioRecordwithAudioPlaybackCaptureConfiguration(Android Q+) - Implemented on-demand audio reading directly from
AudioRecordusingREAD_BLOCKINGmode - Screen audio is read synchronously when requested by the WebRTC audio callback, instead of having two separate threads to read microphone data and screen audio data
Audio Mixing:
- Created
addAndConvertBuffers()utility function to mix microphone and screen audio buffers - Integrated mixing logic in
StreamPeerConnectionFactory.setAudioBufferCallback():- Audio filters are applied to microphone audio first
- Screen audio is then mixed with filtered microphone audio
- When microphone is muted, only screen audio is transmitted
- When microphone is enabled, both are mixed together
Audio Track Management:
- Added
includeAudioparameter toCall.startScreenSharing()(defaults tofalsefor backward compatibility) - Added
audioEnabledStateFlow toScreenShareManagerto track screen share audio state separately from video - When screen sharing starts with
includeAudio=trueand microphone is muted, automatically creates and publishes audio track - When microphone is muted but screen share audio is enabled, keeps audio track published so screen audio can continue
- When screen sharing stops, un-publish audio track if microphone was muted
Key Changes:
-
MediaManager.ScreenShareManager: Added screen audio capture infrastructure andaudioEnabledStateFlow -
StreamPeerConnectionFactory: Added audio mixing logic with proper filter application order -
RtcSession: Updated audio track publishing/unpublishing logic to respect screen share audio state -
MicrophoneManager.disable(): ChecksaudioEnabledto determine if audio track should remain enabled - Demo app: Updated to pass
includeAudio=truewhen starting screen sharing
🎨 UI Changes
None
Testing
On the demo app :
- Start screen sharing with
includeAudio=trueand verify system audio is captured and transmitted - Mute microphone while screen sharing with audio enabled - verify screen audio continues to be heard on the other side
- Unmute microphone while screen sharing - verify both microphone and screen audio are mixed and transmitted
- Join call with microphone muted, then start screen sharing with audio - verify audio track is created automatically
- Stop screen sharing while muted - verify no screen audio is passed to the other side
PR checklist ✅
All required conditions are satisfied:
- Title length is OK (or ignored by label).
- At least one
pr:label exists. - Sections
### Goal,### Implementation, and### Testingare filled.
🎉 Great job! This PR is ready for review.
SDK Size Comparison 📏
| SDK | Before | After | Difference | Status |
|---|---|---|---|---|
| stream-video-android-core | 11.93 MB | 11.93 MB | 0.00 MB | 🟢 |
| stream-video-android-ui-xml | 5.68 MB | 5.70 MB | 0.02 MB | 🟢 |
| stream-video-android-ui-compose | 6.27 MB | 6.27 MB | 0.00 MB | 🟢 |
We are not yet merging this, as there are some discussions around SFU changes.
