google-deepmind
Unity Plugin " The type or namespace name...
Hi,
I'm a Ph.D. student and I'm trying to use MuJoCo for Virtual Reality Sensory Feedback studies. I'm looking for some help with getting an old Haptix model (my lab was the last lab to be in the Haptix program that ended this year)
We are using the LUKE hand model to pick up a block and receive force values off the block to provide sensory feedback to recreate a fragile egg. (if anyone has a model in Unity that already does this, I'd be happy to drop my project and pick up where you have the project)
I've be able to import the XML Below into Unity:
<worldbody>
<geom type="plane" material="MatPlane" pos=".6 0 0.1" zaxis="-1 0 1" size=".1 .5 .025"/>
<geom type="plane" material="MatPlane" pos="-.6 0 0.1" zaxis="1 0 1" size=".1 .5 .025"/>
<geom type="plane" material="MatPlane" pos="0 .6 0.1" zaxis="0 -1 1" size=".5 .1 .025"/>
<geom type="plane" material="MatPlane" pos="0 -.6 0.1" zaxis="0 1 1" size=".5 .1 .025"/>
<body name="obj1" pos="-0.2 0 .1">
<geom type="box" rgba="0 1 0 1" size="0.05 0.03 0.05" density="500"/>
<joint class="free"/>
</body>
<!-- <body name="obj2" pos="0 0 .1"> -->
<!-- <geom type="box" material="MatWoodB" size="0.03 0.03 0.03"/> -->
<!-- <joint class="free"/> -->
<!-- </body> -->
<body name="obj3" pos="0.1 0 .025">
<geom type="box" pos="-0.1 0 0" material="MatWoodG" size="0.01 0.3 0.1"/>
<!-- <geom type="box" pos="0.1 0 0" material="MatWoodG" size="0.01 0.1 0.05"/> -->
<!-- <geom type="box" pos="0 0.1 0" material="MatWoodG" size="0.1 0.01 0.05"/> -->
<!-- <geom type="box" pos="0 -0.1 0" material="MatWoodG" size="0.1 0.01 0.05"/> -->
<!-- <geom type="box" material="MatWoodG" size="0.03 0.03 0.03"/> -->
<!-- <geom type="box" material="MatWoodG" size="0.03 0.03 0.03"/> -->
<!-- <geom type="box" material="MatWoodG" size="0.03 0.03 0.03"/> -->
</body>
</worldbody>
and
<default>
<position ctrllimited="true" forcelimited="true" gear="57.29577951"/>
<geom solimp="0.95 0.99 0.001" solref="0.01 1" friction="1.0 0.01 0.001"/>
<default class="robot">
<geom material="robot" contype="1" conaffinity="2" condim="4"/>
<site rgba=".3 .9 .3 .3" type="box" group="3"/>
<joint limited="true" pos="0 0 0" axis="1 0 0" range="0 90" damping="0.1" armature="0.002"/>
<default class="RP">
<joint damping="0.05" armature="0.001"/>
</default>
</default>
<default class="couple">
<equality solref="0.05 0.5" solimp="0.0001 0.0001 0"/>
</default>
<default class="free">
<joint type="free" damping="0" armature="0" limited="false"/>
</default>
</default>
<statistic extent="0.4" center="0 -0.2 0.2"/>
<visual>
<quality shadowsize="2048"/>
<map fogstart="1" fogend="3" znear="0.05" force="0.002" stiffness="200"/>
<headlight diffuse=".6 .6 .6" specular="0 0 0" active="1"/>
<scale contactwidth="0.1" contactheight="0.03" forcewidth="0.05"/>
</visual>
<include file="assets.xml"/>
<asset>
<mesh name="base" file="Base.STL"/>
<mesh name="forearm" file="Forearm.STL"/>
<mesh name="palm" file="Palm.STL"/>
<mesh name="thumbProximal" file="Thumb_Proximal.STL"/>
<mesh name="thumbDistal" file="Thumb_Distal.STL"/>
<mesh name="thumbNail" file="Thumb_Nail.STL"/>
<mesh name="indexProximal" file="Index_Proximal.STL"/>
<mesh name="indexDistal0" file="Index_Distal0.STL"/>
<mesh name="indexDistal1" file="Index_Distal1.STL"/>
<mesh name="indexNail" file="Index_Nail.STL"/>
<mesh name="middleProximal" file="Middle_Proximal.STL"/>
<mesh name="middleMedian" file="MiddleRing_Median.STL"/>
<mesh name="middleDistal" file="MiddleRing_Distal.STL"/>
<mesh name="middleNail" file="MiddleRing_Nail.STL"/>
<mesh name="ringProximal" file="Ring_Proximal.STL"/>
<mesh name="littleProximal" file="Pinky_Proximal.STL"/>
<mesh name="littleDistal0" file="Pinky_Distal0.STL"/>
<mesh name="littleDistal1" file="Pinky_Distal1.STL"/>
<mesh name="littleNail" file="Pinky_Nail.STL"/>
</asset>
<contact>
<pair geom1="palm" geom2="thumbProximal"/>
</contact>
<worldbody>
<light directional="false" cutoff="35" exponent="10" diffuse=".7 .7 .7" pos="0.4 -0.2 1.5" dir="-0.4 0.2 -1.5" specular=".3 .3 .3"/>
<!-- ======= TABLE ======= -->
<geom name="ground" type="plane" material="table2d" pos="0 0 0" size=".5 .5 .025" condim="1"/>
<geom contype="0" conaffinity="0" type="box" material="tablecube" pos="0 0 -0.026" size=".5 .5 .025"/>
<geom contype="0" conaffinity="0" type="cylinder" fromto=".4 .4 -1.001 .4 .4 -0.002" size="0.04"/>
<geom contype="0" conaffinity="0" type="cylinder" fromto="-.4 .4 -1.001 -.4 .4 -0.002" size="0.04"/>
<geom contype="0" conaffinity="0" type="cylinder" fromto=".4 -.4 -1.001 .4 -.4 -0.002" size="0.04"/>
<geom contype="0" conaffinity="0" type="cylinder" fromto="-.4 -.4 -1.001 -.4 -.4 -0.002" size="0.04"/>
<geom contype="0" conaffinity="0" type="plane" pos="0 0 -1" size="10 10 0.25" material="groundplane"/>
<!-- ======= MOCAP ======= -->
<body name="tracker" mocap="true" pos="0 -0.35 0.2" euler="90 0 0">
<geom type="cylinder" group="1" size="0.04 0.01" contype="0" conaffinity="0" rgba=".9 .5 .5 1"/>
</body>
<!-- ======= ROBOT ======= -->
<body name="base" childclass="robot" pos="0 -0.35 0.2" euler="90 180 30">
<joint name="base" class="free"/>
<geom type="mesh" mesh="base" group="2"/>
<body name="forearm" pos="0 0 0">
<joint name="wristRotation" axis="0 0 -1" armature="0.05" damping="2" range="-120 175" frictionloss="1"/>
<geom type="mesh" mesh="forearm" group="2"/>
<body name="wrist" pos="0 0 0.09255" euler="0 -13 0">
<joint name="wristDeviation" axis="0 1 0" damping="0.2" limited="true" range="-20 20"/>
<geom type="sphere" size="0.01" group="2"/>
<body name="palm" pos="0 0 0" euler="0 0 30">
<joint name="wristFlexion" axis="1.7321 -1 0" armature="0.05" damping="2" range="-55 55" frictionloss="1"/>
<geom name="palm" type="mesh" mesh="palm"/>
<geom type="capsule" fromto="0.038 -0.007 0.045 0.028 -0.003 0.041" size="0.018"/>
<site name="palmProximal" pos="-0.005 -.018 .042" size=".025 .003 .013"/>
<site name="palmDistal" pos="-0.003 -.018 .075" size=".032 .003 .02"/>
<site name="handDorsal" pos="0 .018 .07" size=".035 .003 .025"/>
<site name="handEdge" pos="-0.04 0 .07" size=".003 .015 .025"/>
<!--THUMB-->
<body name="thumbBase" pos="0.0425 -0.01408 0.045">
<joint name="thumbYaw" axis="-1 0 0" range="0 90" frictionloss="0.5"/>
<geom type="sphere" size="0.007"/>
<body name="thumb" pos="0 0 0" euler="0 0 -10">
<joint name="thumbPitch" axis="0 0 -1" range="0 100" frictionloss="0.5"/>
<geom type="capsule" fromto="0.005 0.007 0 0.025 0.007 0" size="0.013" />
<geom name="thumbProximal" type="mesh" mesh="thumbProximal" contype="3"/>
<geom name="thumbDistal" type="mesh" mesh="thumbDistal" contype="3"/>
<geom name="thumbNail" type="mesh" pos="0 0 0.0007" mesh="thumbNail" contype="3" condim="1"/>
<site name="thumbTip" pos="0.07 0.007 0.01" size=".032 .01 .003"/>
<site name="thumbRadial" pos="0.07 -0.002 0" size=".032 .003 .01"/>
<site name="thumbDorsal" pos="0.075 0.007 -0.01" size=".037 .01 .003"/>
<site name="thumbUlnar" pos="0.07 0.016 0" size=".032 .003 .01"/>
</body>
</body>
<!--INDEX-->
<body name="indexProximal" pos="0.0299 -0.00108 0.100">
<joint name="indexProximal"/>
<geom type="mesh" mesh="indexProximal"/>
<body name="indexDistal" pos="0 0 .040">
<joint name="indexDistal"/>
<geom type="mesh" mesh="indexDistal0"/>
<geom name="indexDistal" type="mesh" mesh="indexDistal1"/>
<geom name="indexNail" type="mesh" mesh="indexNail" condim="1"/>
<site name="indexTip" pos="0 -.008 .03" size=".008 .003 .018"/>
<site name="indexLateral" pos="0.008 0 0.028" size=".003 .006 .02"/>
</body>
</body>
<!--MIDDLE-->
<body name="middleProximal" pos="0.0095 -0.00108 0.100">
<joint name="middleProximal"/>
<geom type="mesh" mesh="middleProximal"/>
<body name="middleDistal" pos="0 0 .0443">
<joint name="middleDistal"/>
<geom type="mesh" mesh="middleMedian"/>
<geom type="mesh" mesh="middleDistal" pos="0 0 .025" axisangle="1 0 0 15"/>
<geom name="middleNail" type="mesh" mesh="middleNail" pos="0 0 .025" axisangle="1 0 0 15" condim="1"/>
<site name="middleTip" pos="0 -.01 .03" size=".008 .003 .02"/>
</body>
</body>
<!--RING-->
<body name="ringProximal" childclass="RP" pos="-0.0095 -0.00108 0.100">
<joint name="ringProximal"/>
<geom type="mesh" mesh="ringProximal"/>
<body name="ringDistal" pos="0 0 .03807">
<joint name="ringDistal"/>
<geom type="mesh" mesh="middleMedian"/>
<geom name="ringDistal" type="mesh" mesh="middleDistal" pos="0 0 .025" axisangle="1 0 0 15"/>
<geom name="ringNail" type="mesh" mesh="middleNail" pos="0 0 .025" axisangle="1 0 0 15" condim="1"/>
<site name="ringTip" pos="0 -.01 .03" size=".008 .003 .02"/>
</body>
</body>
<!--LITTLE-->
<body name="littleProximal" childclass="RP" pos="-0.0285 -0.00108 0.09598">
<joint name="littleProximal"/>
<geom type="mesh" mesh="littleProximal"/>
<body name="littleDistal" pos="0 0 .028">
<joint name="littleDistal"/>
<geom type="mesh" mesh="littleDistal0"/>
<geom type="mesh" mesh="littleDistal1"/>
<geom name="littleNail" type="mesh" pos="0 0.001 0" mesh="littleNail" condim="1"/>
<site name="littleTip" pos="0 -.01 .025" size=".008 .003 .015"/>
</body>
</body>
</body>
</body>
</body>
</body>
</worldbody>
<tendon>
<fixed name="index" frictionloss="1">
<joint joint="indexProximal" coef="0.5"/>
<joint joint="indexDistal" coef="0.5"/>
</fixed>
<fixed name="MRP" frictionloss="2">
<joint joint="middleProximal" coef="0.25"/>
<joint joint="middleDistal" coef="0.25"/>
<joint joint="ringProximal" coef="0.125"/>
<joint joint="ringDistal" coef="0.125"/>
<joint joint="littleProximal" coef="0.125"/>
<joint joint="littleDistal" coef="0.125"/>
</fixed>
<fixed name="couple1">
<joint joint="middleProximal" coef="1"/>
<joint joint="middleDistal" coef="-0.2"/>
<joint joint="ringProximal" coef="-0.2"/>
<joint joint="ringDistal" coef="-0.2"/>
<joint joint="littleProximal" coef="-0.2"/>
<joint joint="littleDistal" coef="-0.2"/>
</fixed>
<fixed name="couple2">
<joint joint="middleProximal" coef="-0.2"/>
<joint joint="middleDistal" coef="1"/>
<joint joint="ringProximal" coef="-0.2"/>
<joint joint="ringDistal" coef="-0.2"/>
<joint joint="littleProximal" coef="-0.2"/>
<joint joint="littleDistal" coef="-0.2"/>
</fixed>
<fixed name="couple3">
<joint joint="middleProximal" coef="-0.2"/>
<joint joint="middleDistal" coef="-0.2"/>
<joint joint="ringProximal" coef="1"/>
<joint joint="ringDistal" coef="-0.2"/>
<joint joint="littleProximal" coef="-0.2"/>
<joint joint="littleDistal" coef="-0.2"/>
</fixed>
<fixed name="couple4">
<joint joint="middleProximal" coef="-0.2"/>
<joint joint="middleDistal" coef="-0.2"/>
<joint joint="ringProximal" coef="-0.2"/>
<joint joint="ringDistal" coef="1"/>
<joint joint="littleProximal" coef="-0.2"/>
<joint joint="littleDistal" coef="-0.2"/>
</fixed>
<fixed name="couple5">
<joint joint="middleProximal" coef="-0.2"/>
<joint joint="middleDistal" coef="-0.2"/>
<joint joint="ringProximal" coef="-0.2"/>
<joint joint="ringDistal" coef="-0.2"/>
<joint joint="littleProximal" coef="1"/>
<joint joint="littleDistal" coef="-0.2"/>
</fixed>
<fixed name="couple6">
<joint joint="middleProximal" coef="-0.2"/>
<joint joint="middleDistal" coef="-0.2"/>
<joint joint="ringProximal" coef="-0.2"/>
<joint joint="ringDistal" coef="-0.2"/>
<joint joint="littleProximal" coef="-0.2"/>
<joint joint="littleDistal" coef="1"/>
</fixed>
</tendon>
<equality>
<weld body1="tracker" body2="base" solimp="0.8 0.8 0.001" solref="0.01 1"/>
<joint joint1="wristDeviation" joint2="wristFlexion" polycoef="-0.0057 0.0347 0.0352 -0.1023 0.0993"/>
<joint joint1="indexProximal" joint2="indexDistal"/>
<tendon class="couple" tendon1="couple1"/>
<tendon class="couple" tendon1="couple2"/>
<tendon class="couple" tendon1="couple3"/>
<tendon class="couple" tendon1="couple4"/>
<tendon class="couple" tendon1="couple5"/>
<tendon class="couple" tendon1="couple6"/>
</equality>
<actuator>
<position name="wristRotation" joint="wristRotation" ctrlrange="-175 120" forcerange="-0.3 0.3" kp="0.02"/>
<position name="wristFlexion" joint="wristFlexion" ctrlrange="-55 55" forcerange="-0.3 0.3" kp="0.02"/>
<position name="thumbYaw" joint="thumbYaw" ctrlrange="0 90" forcerange="-0.05 0.05" kp="0.01"/>
<!-- <position name="thumbPitch" joint="thumbPitch" ctrlrange="89 100" forcerange="-0.05 0.05" kp="0.01"/> -->
<position name="thumbPitch" joint="thumbPitch" ctrlrange="0 100" forcerange="-0.05 0.05" kp="0.01"/>
<position name="index" tendon="index" ctrlrange="0 90" forcerange="-0.1 0.1" kp="0.02"/>
<position name="MRP" tendon="MRP" ctrlrange="0 90" forcerange="-0.2 0.2" kp="0.04"/>
</actuator>
<sensor>
<actuatorpos actuator="wristRotation" cutoff="175"/>
<actuatorpos actuator="wristFlexion" cutoff="55"/>
<actuatorpos actuator="thumbYaw" cutoff="90"/>
<actuatorpos actuator="thumbPitch" cutoff="100"/>
<actuatorpos actuator="index" cutoff="90"/>
<actuatorpos actuator="MRP" cutoff="90"/>
<touch site="palmProximal" cutoff="25.5"/>
<touch site="palmDistal" cutoff="25.5"/>
<touch site="handDorsal" cutoff="25.5"/>
<touch site="handEdge" cutoff="25.5"/>
<touch site="thumbTip" cutoff="25.5"/>
<touch site="thumbRadial" cutoff="25.5"/>
<!-- <touch site="thumbDorsal" cutoff="25.5"/> -->
<!-- <touch site="thumbUlnar" cutoff="25.5"/> -->
<touch site="indexTip" cutoff="25.5"/>
<touch site="indexLateral" cutoff="25.5"/>
<touch site="middleTip" cutoff="25.5"/>
<touch site="ringTip" cutoff="25.5"/>
<touch site="littleTip" cutoff="25.5"/>
</sensor>
But I'm running into errors when trying to build and run it.
Other information: I'm running on the Meta Quest Pro I have Unity experience but it's relatively limited in the scope of Unity
Thanks for any and all help! I've tried reading through the unity and mujoco plug in documentation but nothing seems to be pointing out my exact issue.
Thanks again.
The error messages you posted indicate issues in installing the plug-in. You can skip that step and manually place the folder in your Unity asset folder, provided you also fetch the binary and name it as expected (this is mentioned in the documentation IIRC). Did you manage to run any simulation yet? Start with verifying your setup with some simple models first, and then try the Haptix model. I'll leave the big open for now - please let us know how it goes!
After doing the install both ways I'm pretty confident I have the plugin installed correctly. I'm getting the same error after placing the folder into the assets folder manually. I've also tried importing the simple
I don't know what you mean by naming it as expected. I haven't read anything in the Unity section that would imply naming a file by a specific name.
I imported the hammock.xml and tried building with that and got the same error.
Thank you so much for helping! I don't know what steps I've missed as the documentation doesn't mention much other than importing the plug in.
The binary needs to be renamed, search for the word "rename" in the Unity page in the docs.
If you installed everything correctly, you don't need the file MujocoBinaryRetriever.cs and you can remove it -- all it does is automate the binary fetching and renaming when loading the binary the first time.
Try to load something even simpler than hammock -- I recommend the cartpole.
The only "renaming" suggested is for OSX and Linux. I'm on windows so no renaming is required when doing the manual approach.
After deleting MujocoBinaryRetriever.cs I have new errors:
There are almost 70 of these errors. It seems like it just can't find the files even though the unity folder is within the assets folder of the project.
Hello, those errors on build come up when the editor files are included in the build for your platform assembly file. Could you check the assembly file in the Editor folder, and see what platforms are enabled? Everything, except "Editor" should be disabled. This way all scripts that reference editor-only features will be ignored in the build.
This was an issue in an earlier version of the plugin, but I thought it was fixed since then. Is it possible you are using an older version of the plugin?
I just downloaded this plug-in so it should be up to date.
Which editor folder? I found one under... mujocounity-main\unity\Editor
in mujoco.editor.asmdef I have nothing under "includePlatforms": [], "excludePlatforms": [],
Is there a different file I should be looking for?
Thank you all for the help! I've been spending hours trying to figure out my issue.
Hello, it's at MuJoCo/Editor/Mujoco.Editor.asmdef. Could you send a screenshot of how that file looks in your inspector when you click on it in Unity?
Its contents should be this (based on the file on the main branch):
{
"name": "Mujoco.Editor",
"rootNamespace": "",
"references": [
"Mujoco.Runtime"
],
"includePlatforms": [
"Editor"
],
"excludePlatforms": [],
"allowUnsafeCode": true,
"overrideReferences": false,
"precompiledReferences": [],
"autoReferenced": true,
"defineConstraints": [],
"versionDefines": [],
"noEngineReferences": false
}
Interestingly its missing the editor only platform criteria.
Adding it either in a text editor or through the Unity inspector should fix it for you. I'm not sure what could have changed it.
If you do remember the steps you've taken to install could you detail them (starting with "open the MuJoCo repo on github)? I'd like to try to reproduce it and see what could have caused it.
I have done something similar with the MPL hand (real time control and sensing) in Unity + MuJoCo so I think you are on a good path. Let me know if you have any other trouble setting up.
open the MuJoCo repo on github, click the green "<>code" icon, and then download zip.
And then you added the unzipped package via Unity's package manager and the package.json file in unity folder of the repo?
I was able to build and run the flag.xml as well as an old HAPTIX model we are hoping to utilize.
Thank you both, I'm sure I'll be looking for help again. (Is this the best place to ask for help?)