stream-video-android icon indicating copy to clipboard operation
stream-video-android copied to clipboard

Add possibility to share screen audio when screensharing is enabled

Open PratimMallick opened this issue 2 months ago • 4 comments

Goal

This PR adds support for capturing and transmitting screen audio during screen sharing on Android. Previously, when users shared their screen, only the video was transmitted. This change enables capturing system audio (e.g., from media playback, other apps) and mixing it with microphone audio before transmission. Additionally, it ensures that screen audio continues to be transmitted even when the microphone is muted, addressing a common use case where users want to share audio content while remaining muted.

Implementation

Screen Audio Capture:

  • Added screen audio capture using AudioRecord with AudioPlaybackCaptureConfiguration (Android Q+)
  • Implemented on-demand audio reading directly from AudioRecord using READ_BLOCKING mode
  • Screen audio is read synchronously when requested by the WebRTC audio callback, instead of having two separate threads to read microphone data and screen audio data

Audio Mixing:

  • Created addAndConvertBuffers() utility function to mix microphone and screen audio buffers
  • Integrated mixing logic in StreamPeerConnectionFactory.setAudioBufferCallback():
    • Audio filters are applied to microphone audio first
    • Screen audio is then mixed with filtered microphone audio
    • When microphone is muted, only screen audio is transmitted
    • When microphone is enabled, both are mixed together

Audio Track Management:

  • Added includeAudio parameter to Call.startScreenSharing() (defaults to false for backward compatibility)
  • Added audioEnabled StateFlow to ScreenShareManager to track screen share audio state separately from video
  • When screen sharing starts with includeAudio=true and microphone is muted, automatically creates and publishes audio track
  • When microphone is muted but screen share audio is enabled, keeps audio track published so screen audio can continue
  • When screen sharing stops, un-publish audio track if microphone was muted

Key Changes:

  • MediaManager.ScreenShareManager: Added screen audio capture infrastructure and audioEnabled StateFlow
  • StreamPeerConnectionFactory: Added audio mixing logic with proper filter application order
  • RtcSession: Updated audio track publishing/unpublishing logic to respect screen share audio state
  • MicrophoneManager.disable(): Checks audioEnabled to determine if audio track should remain enabled
  • Demo app: Updated to pass includeAudio=true when starting screen sharing

🎨 UI Changes

None

Testing

On the demo app :

  1. Start screen sharing with includeAudio=true and verify system audio is captured and transmitted
  2. Mute microphone while screen sharing with audio enabled - verify screen audio continues to be heard on the other side
  3. Unmute microphone while screen sharing - verify both microphone and screen audio are mixed and transmitted
  4. Join call with microphone muted, then start screen sharing with audio - verify audio track is created automatically
  5. Stop screen sharing while muted - verify no screen audio is passed to the other side

PratimMallick avatar Nov 12 '25 08:11 PratimMallick

PR checklist ✅

All required conditions are satisfied:

  • Title length is OK (or ignored by label).
  • At least one pr: label exists.
  • Sections ### Goal, ### Implementation, and ### Testing are filled.

🎉 Great job! This PR is ready for review.

github-actions[bot] avatar Nov 12 '25 08:11 github-actions[bot]

SDK Size Comparison 📏

SDK Before After Difference Status
stream-video-android-core 11.93 MB 11.93 MB 0.00 MB 🟢
stream-video-android-ui-xml 5.68 MB 5.70 MB 0.02 MB 🟢
stream-video-android-ui-compose 6.27 MB 6.27 MB 0.00 MB 🟢

github-actions[bot] avatar Nov 12 '25 08:11 github-actions[bot]

We are not yet merging this, as there are some discussions around SFU changes.

aleksandar-apostolov avatar Nov 14 '25 09:11 aleksandar-apostolov

Quality Gate Failed Quality Gate failed

Failed conditions
1.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

sonarqubecloud[bot] avatar Nov 27 '25 09:11 sonarqubecloud[bot]