Genesis-Embodied-AI
BUG. Can not visualize texture.
Genesis can not visualize texture for the MJCF model attached.
import genesis as gs
import os
gs.init(backend=gs.cuda)
scene = gs.Scene(show_viewer=True)
mjcf_morph = gs.morphs.MJCF(file=os.path.join('texture_bug', 'floor_room.xml'))
scene.add_entity(mjcf_morph)
scene.build()
Genesis result:
Mujoco (3.2.5) result:
MJCF file with texture image:
MJCF file
<mujoco model="base">
<option impratio="20" cone="elliptic" density="1.2" viscosity="0.00002" timestep="0.002" />
<compiler angle="radian" meshdir="meshes/" inertiagrouprange="0 0" autolimits="true" />
<size nconmax="5000" njmax="5000" />
<asset>
<material texrepeat="2 2" texuniform="true" reflectance="0.1" shininess="0.1" name="floor_room_wall_mat" texture="floor_room_wall" />
<texture type="2d" name="floor_room_wall" file="light_wood_planks_long.png" />
</asset>
<worldbody>
<body pos="2.15 -2.5 -0.02" name="floor_room_main" quat="0.707 0 0 0.707">
<geom conaffinity="0" contype="0" mass="100" material="floor_room_wall_mat" type="box" size="2.54 3.19 0.02" pos="0 0 0" group="1" name="floor_room_g0_vis" />
<site type="sphere" group="0" pos="0 0 0" size="0.002 0.002 0.002" rgba="1 0 0 1" name="floor_room_default_site" />
</body>
</worldbody>
</mujoco>
maybe try this example in rendering folder - see if it imports texture like this ? Screencast from 2024-12-22 12-35-40.webm
maybe try this example in rendering folder - see if it imports texture like this ? Screencast from 2024-12-22 12-35-40.webm
I got the same rendering result as you. The texture is imported fine.
I notice that the texture is rendered fine when it is defined within .obj mesh file. But the texture defined through <texture> element within MJCF file is not rendered properly. Here is the rendering result of a kitchen scene from robocasa. There is ValueError: Incorrect texture coordinate shape and ValueError: Cannot reformat 2-channel texture into RGBA error raised for some objects. I have removed them from scene.
https://github.com/user-attachments/assets/ce67c0ae-96ae-4509-80ad-c0df7c17832b
There are some objects rendered with texture.
The result from mujoco 3.2.5 for the same scene.
@zhouxian 抱歉打扰，请问上面提到的问题是 genesis 对 mjcf 的支持问题，还是我对 texture 的导入方式不对？后续有没有修复或特性支持的计划，谢谢。
抱歉我们暂时人手有些不够issue有点多😅一定会修复的！ 我们会尽可能支持各种mesh和texture格式。（现在对mjcf的支持应该已经比isaac gym好很多了吧😬
如果过两周还没修复再戳我们一下，感谢！
On Mon, Dec 23, 2024 at 8:35 PM Riften @.***> wrote:
@zhouxian https://github.com/zhouxian 抱歉打扰，请问上面提到的问题是 genesis 对 mjcf 的支持问题，还是我对 texture 的导入方式不对？后续有没有修复或特性支持的计划，谢谢。
— Reply to this email directly, view it on GitHub https://github.com/Genesis-Embodied-AI/Genesis/issues/236#issuecomment-2559627661, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEV4V6IKIPBR6FJ3DTPBZ2D2G77QBAVCNFSM6AAAAABUBU4LLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJZGYZDONRWGE . You are receiving this because you were mentioned.Message ID: @.***>
In #288, box texture is loaded inside mjcf parser. At least the wooden plane looks good. Please give it a try and let me know if it works. If so, I will merge it into the main branch.
BTW, @Riften what's the easiest way to get a few mjcf files of robocasa kitchen environment? I want to load it to see if there is anything missing in our mjcf parser
In #288, box texture is loaded inside mjcf parser. At least the wooden plane looks good. Please give it a try and let me know if it works. If so, I will merge it into the main branch.
BTW, @Riften what's the easiest way to get a few mjcf files of robocasa kitchen environment? I want to load it to see if there is anything missing in our mjcf parser
The floor texture seems to be fine.
But there is still some error when load some other texture. The mjcf file which raises error is attached.
Full error message
{
"name": "TypeError",
"message": "Cannot handle this data type: (1, 1, 600), |u1",
"stack": "---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/PIL/Image.py:3098, in fromarray(obj, mode)
3097 try:
-> 3098 mode, rawmode = _fromarray_typemap[typekey]
3099 except KeyError as e:
KeyError: ((1, 1, 600), '|u1')
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
Cell In[5], line 2
1 #mjcf_entity = scene.add_entity(mjcf_morph, surface=file_surface)
----> 2 mjcf_entity = scene.add_entity(mjcf_morph)
3 #obj_entity = scene.add_entity(obj_morph)
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/utils/misc.py:38, in assert_unbuilt.<locals>.wrapper(self, *args, **kwargs)
36 if self.is_built:
37 gs.raise_exception(\"Scene is already built.\")
---> 38 return method(self, *args, **kwargs)
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/engine/scene.py:360, in Scene.add_entity(self, morph, material, surface, visualize_contact, vis_mode)
357 else:
358 morph.decompose_nonconvex = False
--> 360 entity = self._sim._add_entity(morph, material, surface, visualize_contact)
362 return entity
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/engine/simulator.py:134, in Simulator._add_entity(self, morph, material, surface, visualize_contact)
131 entity = self.avatar_solver.add_entity(self.n_entities, material, morph, surface, visualize_contact)
133 elif isinstance(material, gs.materials.Rigid):
--> 134 entity = self.rigid_solver.add_entity(self.n_entities, material, morph, surface, visualize_contact)
136 elif isinstance(material, gs.materials.MPM.Base):
137 entity = self.mpm_solver.add_entity(self.n_entities, material, morph, surface)
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/engine/solvers/rigid/rigid_solver_decomp.py:63, in RigidSolver.add_entity(self, idx, material, morph, surface, visualize_contact)
60 else:
61 entity_class = RigidEntity
---> 63 entity = entity_class(
64 scene=self._scene,
65 solver=self,
66 material=material,
67 morph=morph,
68 surface=surface,
69 idx=idx,
70 idx_in_solver=self.n_entities,
71 link_start=self.n_links,
72 joint_start=self.n_joints,
73 q_start=self.n_qs,
74 dof_start=self.n_dofs,
75 geom_start=self.n_geoms,
76 cell_start=self.n_cells,
77 vert_start=self.n_verts,
78 face_start=self.n_faces,
79 edge_start=self.n_edges,
80 vgeom_start=self.n_vgeoms,
81 vvert_start=self.n_vverts,
82 vface_start=self.n_vfaces,
83 visualize_contact=visualize_contact,
84 )
85 self._entities.append(entity)
87 return entity
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/engine/entities/rigid_entity/rigid_entity.py:68, in RigidEntity.__init__(self, scene, solver, material, morph, surface, idx, idx_in_solver, link_start, joint_start, q_start, dof_start, geom_start, cell_start, vert_start, face_start, edge_start, vgeom_start, vvert_start, vface_start, visualize_contact)
64 self._visualize_contact = visualize_contact
66 self._is_built = False
---> 68 self._load_model()
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/engine/entities/rigid_entity/rigid_entity.py:78, in RigidEntity._load_model(self)
75 self._load_mesh(self._morph, self._surface)
77 elif isinstance(self._morph, gs.morphs.MJCF):
---> 78 self._load_MJCF(self._morph, self._surface)
80 elif isinstance(self._morph, gs.morphs.URDF):
81 self._load_URDF(self._morph, self._surface)
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/engine/entities/rigid_entity/rigid_entity.py:342, in RigidEntity._load_MJCF(self, morph, surface)
339 world_g_info.append(g_info)
341 else:
--> 342 g_info = mju.parse_geom(mj, i_g, morph.scale, morph.convexify, surface, morph.file)
343 if g_info is not None:
344 link_idx = mj.geom_bodyid[i_g] - 1
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/genesis/utils/mjcf.py:306, in parse_geom(mj, i_g, scale, convexify, surface, xml_path)
304 tex_repeat = mj.mat_texrepeat[mat_id].astype(int)
305 image_array = np.tile(image_array, (tex_repeat[0], tex_repeat[1], 1))
--> 306 visual = TextureVisuals(uv=uv_coordinates, image=Image.fromarray(image_array))
307 tmesh.visual = visual
309 elif mj_type == mujoco.mjtGeom.mjGEOM_MESH:
File ~/App/anaconda3/envs/robotic/lib/python3.9/site-packages/PIL/Image.py:3102, in fromarray(obj, mode)
3100 typekey_shape, typestr = typekey
3101 msg = f\"Cannot handle this data type: {typekey_shape}, {typestr}\"
-> 3102 raise TypeError(msg) from e
3103 else:
3104 rawmode = mode
TypeError: Cannot handle this data type: (1, 1, 600), |u1"
}
As for the mjcf files of robocasa kitchen environment, I have written some export code to get them.
robocasa_exporter.py
import os
import robosuite
from robosuite.controllers import load_composite_controller_config
from robocasa.environments.kitchen.kitchen import Kitchen
import xml.etree.ElementTree as ET
from typing import List
import xml.etree.ElementTree as ET
from typing import List, Dict
# Author: Riften https://github.com/Riften
# MJCF reference https://mujoco.readthedocs.io/en/stable/XMLreference.html
def indent(elem: ET.Element, level=0):
i = "\n" + level*" "
j = "\n" + (level-1)*" "
if len(elem):
if not elem.text or not elem.text.strip():
elem.text = i + " "
if not elem.tail or not elem.tail.strip():
elem.tail = i
for subelem in elem:
indent(subelem, level+1)
if not elem.tail or not elem.tail.strip():
elem.tail = j
else:
if level and (not elem.tail or not elem.tail.strip()):
elem.tail = j
return elem
class MJCFHeader:
"""
A MJCF header contains the bacis xml elements of a mujoco model.
"""
def __init__(self, xml_str: str):
root = ET.fromstring(xml_str)
if root.tag != 'mujoco':
raise ValueError('Root tag of MJCFHeader must be <mujoco>')
self.option = root.find('option')
self.compiler = root.find('compiler')
self.size = root.find('size')
self.statistic = root.find('statistic')
# self.actuator = root.find('actuator')
for child in list(root):
root.remove(child)
self.root = root
def build(self):
new_root = ET.Element(self.root.tag)
new_root.attrib['model'] = self.root.attrib.get('model', '')
for element in ['option', 'compiler', 'size', 'statistic']:
elem = getattr(self, element)
if elem is not None:
new_root.append(elem)
return new_root
class MJCFActuator:
def __init__(self, xml_str: str):
root = ET.fromstring(xml_str)
if root.tag != 'actuator':
raise ValueError('Root tag of MJCFActuator must be <actuator>')
self.root = root
self.joint_actuators = {}
for child in list(root):
joint_name = child.attrib.get('joint')
self.joint_actuators[joint_name] = child
class MJCFAsset:
def __init__(self, xml_str: str):
root = ET.fromstring(xml_str)
if root.tag != 'asset':
raise ValueError('Root tag of MJCFAsset must be <asset>')
self.root = root
self.textures = {}
self.materials = {}
self.meshs = {}
for child in list(root):
if child.tag == 'texture':
if 'name' in child.attrib:
#raise ValueError('Texture element must have a name attribute')
self.textures[child.attrib['name']] = child
else:
print('Texture element does not have a name attribute')
elif child.tag == 'material':
self.materials[child.attrib['name']] = child
elif child.tag == 'mesh':
self.meshs[child.attrib['name']] = child
def retrive_body_requirement(self,
body: ET.Element,
required_materials: Dict[str, ET.Element],
required_textures: Dict[str, ET.Element],
required_meshs: Dict[str, ET.Element],
):
for child in list(body):
if child.tag == 'body':
self.retrive_body_requirement(child, required_materials, required_textures, required_meshs)
if child.tag == 'geom':
material = child.attrib.get('material')
if material is not None:
if material not in required_materials:
# check if the material is defined in the assets
if material not in self.materials:
print(f'Material {material} is not defined in assets.')
required_materials[material] = self.materials[material]
texture = self.materials[material].attrib.get('texture')
if texture is not None:
if texture not in required_textures:
if texture not in self.textures:
print(f'Texture {texture} is not defined in assets.')
required_textures[texture] = self.textures[texture]
mesh = child.attrib.get('mesh')
if mesh is not None:
if mesh not in required_meshs:
if mesh not in self.meshs:
print(f'Mesh {mesh} is not defined in assets.')
required_meshs[mesh] = self.meshs[mesh]
def retrive_body_requirement_recursive(body: ET.Element,
required_materials: List[str],
):
"""
Recursively retrive the body requirements from the body element.
"""
for child in list(body):
if child.tag == 'body':
retrive_body_requirement_recursive(child, required_materials)
if child.tag == 'geom':
material = child.attrib.get('material')
if material is not None:
required_materials.append(material)
def remove_geom_group_0(root: ET.Element):
for child in list(root):
if child.tag == 'geom':
if 'group' in child.attrib and child.attrib['group'] == '0':
root.remove(child)
else:
remove_geom_group_0(child)
def build_mjcf_from_body_elements(body_elements: List[ET.Element],
header: MJCFHeader,
assets: MJCFAsset,
actuator: MJCFActuator = None,
remove_group_0 = True
):
root = header.build()
required_materials: Dict[str, ET.Element] = {}
required_textures: Dict[str, ET.Element] = {}
required_meshs: Dict[str, ET.Element] = {}
for body in body_elements:
assets.retrive_body_requirement(body, required_materials, required_textures, required_meshs)
asset_element = ET.Element('asset')
for material in required_materials.values():
asset_element.append(material)
for texture in required_textures.values():
asset_element.append(texture)
for mesh in required_meshs.values():
asset_element.append(mesh)
root.append(asset_element)
if actuator is not None:
root.append(actuator.root)
worldbody = ET.Element('worldbody')
for body in body_elements:
worldbody.append(body)
root.append(worldbody)
if remove_group_0:
remove_geom_group_0(root)
return root
_UNSUPPORTED_FIXTURES = {
'paper_towel_main_group': 'ValueError: Incorrect texture coordinate shape',
'outlet_room': 'ValueError: Cannot reformat 2-channel texture into RGBA',
'outlet_2_room': 'ValueError: Cannot reformat 2-channel texture into RGBA',
'coffee_machine_main_group': 'ValueError: Cannot reformat 2-channel texture into RGBA'
}
def load_robocasa_env(layout: int=1, style: int=1) -> Kitchen:
config = {
"env_name": "PnPCounterToCab",
"robots": "PandaOmron",
"controller_configs": load_composite_controller_config(robot="PandaOmron"),
"translucent_robot": False,
}
# there is default layout and texture
env: Kitchen = robosuite.make(
**config,
has_renderer=True,
has_offscreen_renderer=False,
render_camera=None,
ignore_done=True,
use_camera_obs=False,
control_freq=20,
renderer="mjviewer",
)
env.layout_and_style_ids = [[layout, style]]
_ = env.reset()
return env
def set_group_transparency(root: ET.Element, transparent_group: List[int] = [0], alpha: float = 0.0):
for geom in root.findall('.//geom'):
group = int(geom.get('group', -1))
if group in transparent_group:
geom.set('rgba', f'1 1 1 {alpha}')
def export_fixtures_to_dir(env: Kitchen, out_dir: str,
transparent_group: List[int] = [0]):
env_xml_str = env.model.get_xml()
root = ET.fromstring(env_xml_str)
asset_element = root.find('asset')
header = MJCFHeader(env_xml_str)
asset = MJCFAsset(ET.tostring(asset_element))
for fixture_name in env.fixtures:
if fixture_name in _UNSUPPORTED_FIXTURES:
print(f"Skipping {fixture_name} due to {_UNSUPPORTED_FIXTURES[fixture_name]}")
continue
fixture = env.fixtures[fixture_name]
fixture_mjcf = build_mjcf_from_body_elements([fixture.get_obj()], header, asset,
remove_group_0 = False)
set_group_transparency(fixture_mjcf, transparent_group, alpha=0.0)
with open(os.path.join(out_dir, f'{fixture_name}.xml'), 'w') as f:
f.write(ET.tostring(indent(fixture_mjcf), encoding='unicode'))
if __name__ == '__main__':
env = load_robocasa_env(1,1)
os.makedirs('fixtures', exist_ok=True)
export_fixtures_to_dir(env, 'fixtures')
Feel free to use it.
I fixed PR #288 to support gray images as textures. Please give it a try. Thanks for providing the code. I randomly loaded 5 fixtures, and it looks good to me. Let me know if there are any further issues.
I fixed PR #288 to support gray images as textures. Please give it a try. Thanks for providing the code. I randomly loaded 5 fixtures, and it looks good to me. Let me know if there are any further issues.
Textures are loaded, while the position and scale of geoms seems to be wrong.
before
https://github.com/user-attachments/assets/15752613-e33f-4b1b-90fa-f6eb90cf30b7
now
https://github.com/user-attachments/assets/688786bc-bb7d-44a6-9920-0d0114cc31d9
I just load all the fixtures exported.
def load_mjcf_dir(scene: gs.Scene, load_dir: str):
for mjcf_filename in os.listdir(load_dir):
if not mjcf_filename.endswith('.xml'):
continue
mjcf_morph = gs.morphs.MJCF(file=os.path.join(load_dir, mjcf_filename))
scene.add_entity(mjcf_morph)
It seems that the box created has different size than before. Here is the vertices coordinate for mjcf in texture_bug.zip
scene = gs.Scene(show_viewer=True)
mjcf_morph = gs.morphs.MJCF(file=os.path.join('texture_bug', 'floor_room.xml'))
mjcf_entity = scene.add_entity(mjcf_morph)
scene.build()
mjcf_entity.vgeoms[0].get_trimesh().vertices
result before
TrackedArray([[-2.54, -3.19, -0.02],
[-2.54, -3.19, 0.02],
[-2.54, 3.19, -0.02],
[-2.54, 3.19, 0.02],
[ 2.54, -3.19, -0.02],
[ 2.54, -3.19, 0.02],
[ 2.54, 3.19, -0.02],
[ 2.54, 3.19, 0.02]])
result now
TrackedArray([[ 0. , 0. , -0.02],
[ 0. , 0. , 0.02],
[ 0. , 1. , -0.02],
[ 0. , 1. , 0.02],
[ 1. , 0. , -0.02],
[ 1. , 0. , 0.02],
[ 1. , 1. , -0.02],
[ 1. , 1. , 0.02]])
It works! Thank you very much for the timely fix! I will close this issue shortly.