Eval with mps gives bad results

[Xuzzo]

System Info

- `lerobot` version: 0.1.0
- Platform: macOS-14.7.1-arm64-arm-64bit
- Python version: 3.10.15
- Huggingface_hub version: 0.26.2
- Dataset version: 3.1.0
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.5.1 (False)
- Cuda version: N/A
- Using GPU in script?: <fill in>

Information

• [ ] One of the scripts in the examples/ folder of LeRobot
• [ ] My own task or dataset (give details below)

Reproduction

The diffusion_pusht agent from running

python lerobot/scripts/eval.py \
    -p lerobot/diffusion_pusht \
    eval.n_episodes=10 \
    eval.batch_size=10

fails all eval episodes on mac. After some digging, this is coming from pytorch/pytorch#139550

Expected behavior

The same agent works with Cuda