rendervulkan: check max extents for format modifier

[njdom24]

Adds a check for the format modifier's extents. Previous behavior would fail silently and render a black screen from an unsuitable format being selected.

Immediately noticeable from trying to use a 4k output on Sway with WLR_RENDERER=vulkan on RDNA <= 3 (tested on 5700 XT).

Resolves https://github.com/ValveSoftware/gamescope/issues/1977, https://github.com/ValveSoftware/gamescope/issues/2013