Is real-time visualization possible in Google Colab? Attempted frame-by-frame image conversion as an alternative

[ardkyer]

Issue Description

When running Genesis in Google Colab environment, real-time visualization is not possible due to OpenGL restrictions. Using show_viewer=True option results in an error.

Alternative Approach

I implemented visualization by saving images frame by frame. Here's a minimal working example:

import genesis as gs
import numpy as np
from PIL import Image
import os

gs.init(backend=gs.cuda)

# Create a directory to save the results
save_dir = "/content/simulation_frames"
os.makedirs(save_dir, exist_ok=True)

scene = gs.Scene(show_viewer=False)

# Add objects
plane = scene.add_entity(gs.morphs.Plane())
box = scene.add_entity(
    gs.morphs.Box(
        pos=(0, 0, 2),
        size=(0.5, 0.5, 0.5)
    )
)

# Camera settings
cam = scene.add_camera(
    res=(320, 240),
    pos=(2.0, 2.0, 2.0),
    lookat=(0, 0, 0),
    fov=30,
)

scene.build()

# Render and save images
for i in range(30):
    scene.step()
    if i % 5 == 0:
        print(f"Step {i}/30")
        rgb_data = cam.render(rgb=True)
        
        # rgb_data[0] contains the actual image data
        img_array = rgb_data[0]
        img = Image.fromarray((img_array * 255).astype(np.uint8))
        img.save(f"{save_dir}/frame_{i:03d}.png")
        print(f"Frame {i} saved to {save_dir}")

Results

While real-time visualization is not possible, frame-by-frame image saving works successfully FPS varies by environment, but achieves about 10-15 FPS on A100 GPU Saved images can be found in the '/content/simulation_frames' directory

Questions

Are there any other methods to achieve real-time visualization in Colab? #166 Are there ways to improve the performance of the current approach? Is real-time visualization possible in other cloud environments?

Environment Details

Google Colab GPU: NVIDIA A100-SXM4-40GB

[Logic-TARS]

I have the same issue. Is it possible to use SSH to access an interactive window on a remote server? From my tests, it seems not to work. I ran the same installation program on both a remote server and a local server. The local server with a display had no issues, but the remote server couldn't show the interactive window.

[ardkyer]

I have the same issue. Is it possible to use SSH to access an interactive window on a remote server? From my tests, it seems not to work. I ran the same installation program on both a remote server and a local server. The local server with a display had no issues, but the remote server couldn't show the interactive window.

I'm not sure, but I think it's a different problem because it's ambiguous to view the remote server and COLab as the same environment. No, it could be the same problem.

[wangyian-me]

Met with the same problem before. Have always been using Genesis on clusters without a monitor. In my experience, saving images frames by frames could be slow because of the IO. I instead save all images into an mp4 by using cv2.VideoWriter, which could save some times.

[Taka-Hashimoto]

In the Google Colab environment with a T4 GPU, I tried the method described in this comment, which significantly improved the FPS to about 50.

[DTSwink]

would someone share their collab genesis installation ? did it on my machine but struggle on collab.. @Taka-Hashimoto

[DTSwink]

actually it worked using !git clone https://github.com/Genesis-Embodied-AI/Genesis.git :)