Update Protobuf Version for Compatibility

[hesic73]

We currently build Deoxys using Protobuf 3. However, using a newer version of Protobuf in Python (e.g., protobuf==5.28.3) causes compatibility issues.

https://github.com/UT-Austin-RPL/deoxys_control/blob/97396fd91324e9e961f061544e80a208889526ff/deoxys/InstallPackage#L45

$ python examples/run_deoxys_with_space_mouse.py 
Traceback (most recent call last):
  File "examples/run_deoxys_with_space_mouse.py", line 4, in <module>
    from deoxys.franka_interface import FrankaInterface
  File "/home/abrar/hsc/deoxys_control/deoxys/deoxys/franka_interface/__init__.py", line 1, in <module>
    from .franka_interface import FrankaInterface
  File "/home/abrar/hsc/deoxys_control/deoxys/deoxys/franka_interface/franka_interface.py", line 10, in <module>
    import deoxys.proto.franka_interface.franka_controller_pb2 as franka_controller_pb2
  File "/home/abrar/hsc/deoxys_control/deoxys/deoxys/proto/franka_interface/franka_controller_pb2.py", line 36, in <module>
    _descriptor.EnumValueDescriptor(
  File "/home/abrar/miniconda3/envs/hsc/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 920, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

For now, I regenerate the Python files using protoc, which seems to resolve the issue. Here’s the command I use:

python -m grpc_tools.protoc -I=proto --python_out=deoxys/proto --grpc_python_out=deoxys/proto/ proto/franka-interface/*.proto
python -m grpc_tools.protoc -I=proto --python_out=deoxys/proto --grpc_python_out=deoxys/proto proto/*.proto

[hesic73]

By the way, could you add an uninstall target in the Makefile? When I first ran

make -j build_deoxys=1

I built protobuf from source, but I used the default version (3.13.0). Now, I’d like to switch to a newer version. However, after checking out the desired protobuf version and re-running this command, it didn't update the installation as expected. The functionality is unaffected, but having an uninstall target would make it easier to manage version updates.

[AIRJASON50]

Hi @hesic73,

I noticed you encountered a similar issue with protobuf version conflicts when using deoxys_control (mentioned in your original post and comments). I’m facing a related problem and would love to hear how you resolved it!

I’m trying to reproduce a project: seebelow, which specifies protobuf==3.20.0 in its setup.py. However, deoxys defaults to protobuf v3.13.0 (as seen in the InstallPackage script with git checkout v3.13.0). This version mismatch is causing conflicts for me. Did you manage to resolve this kind of versioning issue in your setup? If so, how did you handle it?

I’m also curious about the potential consequences of such a conflict. Could it lead to runtime errors or other issues in the code? If I were to update deoxys to use protobuf 3.20.0 or even a newer version like 5.x.x (by regenerating the proto files, for example), what impacts might I expect—e.g., compatibility breaks or unexpected behavior?

Any insights from your experience would be super helpful. Thanks so much for sharing!

您好 @hesic73，

我在您之前的问题和评论中注意到，您也遇到了与 protobuf 版本冲突类似的问题。我现在面临一个相关的问题，很想知道您是如何解决的！

我正在尝试复现一个项目：seebelow，它的 setup.py 中指定了 protobuf==3.20.0。然而，deoxys 默认使用 protobuf v3.13.0（在 InstallPackage 脚本中通过 git checkout v3.13.0 指定），这种版本不一致给我带来了冲突。请问您在自己的环境中是如何解决这类版本问题的？

我还想了解这种冲突可能带来的后果。它会导致代码无法运行，还是会有其他影响？如果我直接将 deoxys 更新到 protobuf 3.20.0 或更高版本（如 5.x.x，例如通过重新生成 proto 文件），会有什么后果？比如兼容性问题或意外行为？

非常感谢您能分享您的经验，任何建议都对我帮助很大！

[hesic73]

I think the biggest issue with deoxys is that it installs protobuf 3.13.0 directly at the system level, which can easily cause conflicts with other libraries.

Here’s how I worked around it: instead of installing protobuf 3.13.0 system-wide, I installed it in a custom path. You can refer to this thread for details: https://groups.google.com/g/protobuf/c/3CxS0Gb7tM8 or just ask ChatGPT.

Then, following the instructions in issue #28, I added the following before compiling:

export LIBRARY_PATH=/home/abrar/hsc/protobuf-3.13.0-install/lib:$LIBRARY_PATH

And when running the deoxys binary (e.g., the auto_arm.sh script), I prepended the following:

export LD_LIBRARY_PATH=/home/abrar/hsc/protobuf-3.13.0-install/lib:$LD_LIBRARY_PATH
./auto_scripts/auto_arm.sh <config path>

Also, in principle, I don't think it's strictly necessary to use protobuf 3.13.0 to build deoxys. You could try using the system’s default version of protobuf, then in the InstallPackage script, check out the corresponding compatible version of the library. I haven’t tested this myself, but it might work.

which protoc
protoc --version

[jstmn]

For anyone else running into this, pip install protobuf==3.20.0 worked from me. This was suggested in this comment: https://github.com/UT-Austin-RPL/deoxys_control/issues/10#issuecomment-1832628328