Sunshine icon indicating copy to clipboard operation
Sunshine copied to clipboard
Published 3 weeks ago • verified publisher icon LizardByte

Cross-device DMA-BUF sharing for VAAPI with KMS grab

Open cgutman opened this issue 2 years ago • 8 comments

Description

This introduces support for cross-device VAAPI encoding without having to copy each frame from the capture adapter through RAM to the encoding adapter. If the encoding device can successfully import the DMA-BUF directly from the capture device, we will pass FBs between the adapters.

In the event that the capture device doesn't support VAAPI encoding (which is the case for VKMS), we will try to find another device that can successfully import the FBs from the capture device for encoding. This allows VKMS streaming to work as long as the render device (if any) support VAAPI. Software encoding with VKMS with imported DMA-BUFs from a primary render GPU is still unsupported.

Screenshot

Issues Fixed or Closed

Fixes #2044 for GPU encoding only

Type of Change

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Dependency update (updates to dependencies)
  • [ ] Documentation update (changes to documentation)
  • [ ] Repository update (changes to repository files, e.g. .github/...)

Checklist

  • [x] My code follows the style guidelines of this project
  • [x] I have performed a self-review of my own code
  • [x] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

Branch Updates

LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch must be updated before it can be merged. You must also Allow edits from maintainers.

  • [x] I want maintainers to keep my branch updated

cgutman avatar Jan 21 '24 02:01 cgutman

Just tested this out and this indeed fixes #2044 for me.

SzczurekYT avatar Jan 23 '24 22:01 SzczurekYT

I tested it and it appears to work for me as well with the exception of the connection seemingly terminating at random. Perhaps once this is rebased on the the latest release, then I can test it again to make sure that it wasn't some other part of the code base causing the problem.

d-air1 avatar Mar 05 '24 06:03 d-air1

This is interesting. I checked out the PR, but unfortunately I could note compile it:

build$ cmake .. && make -j12 && cpack -G DEB
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
(...)
--   Found libcurl, version 7.81.0
-- Checking for module 'miniupnpc'
--   Found miniupnpc, version 2.2.3
CMake Error at cmake/dependencies/common.cmake:23 (add_subdirectory):
  The source directory

    /home/ubuntu/Sunshine/third-party/nlohmann_json

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:46 (include)


-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found version "1.74.0") found components: locale log filesystem program_options 
-- Looking for a CUDA compiler
(...)

Somehow the nlohman_json folder is empty, but the other directories under third-party are populated. A recursive git pull fetches multiple submodules, but not the nlohman_json one.

gschintgen avatar May 05 '24 19:05 gschintgen

tl:dr: Couldn't import RGB Image: 00003009

I manually cloned the json library to give it another go. I'd just like to leave some feedback, in case this is unexpected behavior: With an AMD dGPU (RX6650XT) as card1/renderD129 and an Intel iGPU (card0/D128) cross-encoding does not work (display/render on AMD, encode on Intel). This is on a regular desktop PC, not a highly integrated dual-GPU-notebook. Here's the log in case it's helpful:

[2024:05:11:20:38:10]: Info: Sunshine version: 0.21.0.faf6bb0.dirty
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card0 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card1 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: crtc(0, 0)
[2024:05:11:20:38:10]: Debug: crtc(2560, 1440)
[2024:05:11:20:38:10]: Debug: plane->possible_crtcs == 1
[2024:05:11:20:38:10]: Debug: x(0) y(0) crtc_x(0) crtc_y(0) crtc_id(85)
[2024:05:11:20:38:10]: Debug: Resolution: 2560x1440: Pitch: 10240: Offset: 0
[2024:05:11:20:38:10]: Debug: Format [XR24, AR24, RA24, XR30, XB30, AR30, AB30, XR48, XB48, AR48, AB48, XB24, AB24, RG16, NV12, P010, XR4H, AR4H, XB4H, AB4H]
[2024:05:11:20:38:10]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2024:05:11:20:38:10]: Debug: Monitor description
[2024:05:11:20:38:10]: Debug: Resolution: 0x0
[2024:05:11:20:38:10]: Debug: Offset: 0x0
[2024:05:11:20:38:10]: Debug: Monitor description
[2024:05:11:20:38:10]: Debug: Resolution: 0x0
[2024:05:11:20:38:10]: Debug: Offset: 0x0
[2024:05:11:20:38:10]: Debug: Monitor description
[2024:05:11:20:38:10]: Debug: Resolution: 2560x1440
[2024:05:11:20:38:10]: Debug: Offset: 0x0
[2024:05:11:20:38:10]: Debug: Desktop resolution: 2560x1440
[2024:05:11:20:38:10]: Info: Trying encoder [vaapi]
[2024:05:11:20:38:10]: Info: Screencasting with KMS
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card0 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Info: Using manually selected adapter for encoding: /dev/dri/card1 -> /dev/dri/renderD128
[2024:05:11:20:38:10]: Debug: Opening render node: /dev/dri/renderD128
[2024:05:11:20:38:10]: Info: Found monitor for DRM screencasting
[2024:05:11:20:38:10]: Debug: crtc(0, 0)
[2024:05:11:20:38:10]: Debug: crtc(2560, 1440)
[2024:05:11:20:38:10]: Debug: plane->possible_crtcs == 1
[2024:05:11:20:38:10]: Debug: x(0) y(0) crtc_x(0) crtc_y(0) crtc_id(85)
[2024:05:11:20:38:10]: Debug: Resolution: 2560x1440: Pitch: 10240: Offset: 0
[2024:05:11:20:38:10]: Debug: Format [XR24, AR24, RA24, XR30, XB30, AR30, AB30, XR48, XB48, AR48, AB48, XB24, AB24, RG16, NV12, P010, XR4H, AR4H, XB4H, AB4H]
[2024:05:11:20:38:10]: Info: Found connector ID [103]
[2024:05:11:20:38:10]: Info: Found cursor plane [82]
...
[2024:05:11:20:38:10]: Debug: EGL: [Mesa Project]: version [1.5]
[2024:05:11:20:38:10]: Debug: API's supported: [OpenGL OpenGL_ES ]
[2024:05:11:20:38:10]: Debug: GL: vendor: Intel
[2024:05:11:20:38:10]: Debug: GL: renderer: Mesa Intel(R) UHD Graphics 770 (ADL-S GT1)
[2024:05:11:20:38:10]: Debug: GL: version: 4.6 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
[2024:05:11:20:38:10]: Debug: GL: shader: 4.60
[2024:05:11:20:38:10]: Info: SDR color coding [Rec. 601]
[2024:05:11:20:38:10]: Info: Color depth: 8-bit
[2024:05:11:20:38:10]: Info: Color range: [JPEG]
[2024:05:11:20:38:10]: Info: vaapi vendor: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ()
[2024:05:11:20:38:10]: Debug: [AVHWDeviceContext @ 0x559fc8e67040] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ().
[2024:05:11:20:38:10]: Debug: [AVHWDeviceContext @ 0x559fc8e67040] Driver not found in known nonstandard list, using standard behaviour.
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Input surface format is nv12.
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Using VAAPI profile VAProfileH264High (7).
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Using VAAPI entrypoint VAEntrypointEncSliceLP (8).
[2024:05:11:20:38:10]: Debug: [h264_vaapi @ 0x559fc8dbf980] Using VAAPI render target format YUV420 (0x1).
...
[2024:05:11:20:38:10]: Info: Found H.264 encoder: h264_vaapi [vaapi]
...
[2024:05:11:20:38:10]: Info: Found HEVC encoder: hevc_vaapi [vaapi]
...
[2024:05:11:20:38:25]: Info: CLIENT CONNECTED
...
[2024:05:11:20:38:25]: Debug: [AVHWDeviceContext @ 0x7f3a581aebc0] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ().
[2024:05:11:20:38:25]: Debug: [AVHWDeviceContext @ 0x7f3a581aebc0] Driver not found in known nonstandard list, using standard behaviour.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Input surface format is nv12.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using VAAPI profile VAProfileH264High (7).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using VAAPI entrypoint VAEntrypointEncSliceLP (8).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using VAAPI render target format YUV420 (0x1).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC mode: CBR.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC target: 100% of 20000000 bps over 1000 ms.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC buffer: 20000000 bits, initial fullness 15000000 bits.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] RC framerate: 60/1 (60.00 fps).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Driver does not report any additional prediction constraints.
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using intra and P-frames (supported references: 3 / 0).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] All wanted packed headers available (wanted 0xd, found 0x1f).
[2024:05:11:20:38:25]: Debug: [h264_vaapi @ 0x7f3a581057c0] Using level 4.2.
[2024:05:11:20:38:25]: Error: Couldn't import RGB Image: 00003009
[2024:05:11:20:38:25]: Error: Could not convert image
...

gschintgen avatar May 11 '24 19:05 gschintgen