any updates on visualization in remote server?

[jhchong0405]

Hi I am having this issue while setting up SAPIEN simulator in VNC server on a remote server.

File "/root/miniconda3/envs/dexart2/lib/python3.8/site-packages/sapien/example/hello_world.py", line 38, in main viewer = Viewer(renderer) # Create a viewer (window) File "/root/miniconda3/envs/dexart2/lib/python3.8/site-packages/sapien/utils/viewer.py", line 223, in init self.window = self.renderer.create_window( RuntimeError: vk::PhysicalDevice::getSurfaceFormatsKHR: ErrorUnknown

I read through the issues and am aware that you mentioned in Aug 2021 that you don't recommend doing SAPIEN visualization remotely, I would like to know if there is any updates to this issue. Is there any way to set up SAPIEN simulator for visualization in VNC server for remote server now (Jan 2024)?

"The first thing you should try is setting the environment variable VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json (make sure the file exists). If other issues happen with present surfaces, you can then disable display. visualization is very hard to set up on a remote server and there are specific VNC parameters required. I recommend not doing visualization remotely."

Originally posted by @fbxiang in https://github.com/haosulab/SAPIEN/issues/34#issuecomment-902818534

[fbxiang]

Hi, I have extensively tested remote visualization recently and here are the findings. First, Vulkan does not officially support any kind of X-forwarding, so I cannot guarantee that remote visualization will work. (It can depend on many factors, including OS version, driver version, GPU vendor, and luck.) However, if you do not need on-screen rendering (i.e. open a window), Vulkan is very reliable.

If you are lucky, the following steps will make VNC to work.

Offscreen rendering on a server

To use SAPIEN on a GPU server without display, the only system dependencies required are libegl1 and libxext6. If using NVIDIA docker environment, enable graphics, utility, and compute by setting the environment variable in the Dockerfile.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute

Virtual desktop on a server

To use SAPIEN on a GPU server with virtual display, additionally install xvfb, x11vnc, and any window manager such as fluxbox or xfce. Add display capabilities for NVIDIA docker.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute,display

Assuming fluxbox, start a VNC server by

x11vnc -create -env FD_PROG=/usr/bin/fluxbox  -env X11VNC_FINDDISPLAY_ALWAYS_FAILS=1 -env X11VNC_CREATE_GEOM=${99:-1920x1080x16} -gone 'pkill Xvfb' -nopw
# Note: you should only allow local access and tunnel through ssh

Now you can connect to the server at port 5900. SAPIEN should be fully functional, test with

python -m sapien.exapmle.hello_world

[jhchong0405]

Thank you for your helpful reply! If you have any updates on remote visualization, please let me know!

[lucywang720]

Hi, if i just want it to generate a video instead of matplotlib rendering, is this issue suitable for my problem?