ros
Improper duplicate field exception?
I must be doing something wrong, or would such a bug go unnoticed from the beginning of rosmsg?
I have a custom message which works with all the CLI tools and C++ (rosbag play, roscpp pubsub, rosmsg show). I can even load it in python using from darpa_msgs.msg import LocalizationHypotheses. But I can't get it to work with rosbag python module.
$ python
>>> import rosbag
>>> b = rosbag.Bag("robot_data_0_TEAMBASE.bag", "r")
>>> msg = next(b.read_messages(topics=["/robot_data/X1/confirmed_hypotheses_batch"]))[1]
WARNING: For type [darpa_msgs/LocalizationHypotheses] stored md5sum [7da59d174d4c697d0571b3a6fc9fed73] has invalid message definition."
>>> import genpy
>>> import rosbag.bag
>>> old = rosbag.bag._get_message_type
>>> def tmp(info):
... genpy.dynamic.generate_dynamic(info.datatype, info.msg_def)[info.datatype]
... return old(info)
...
>>> rosbag.bag._get_message_type = tmp
>>> msg = next(b.read_messages(topics=["/robot_data/X1/confirmed_hypotheses_batch"]))[1]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/ros/melodic/lib/python2.7/dist-packages/rosbag/bag.py", line 2699, in read_messages
yield self.seek_and_read_message_data_record((entry.chunk_pos, entry.offset), raw, return_connection_header)
File "/opt/ros/melodic/lib/python2.7/dist-packages/rosbag/bag.py", line 2862, in seek_and_read_message_data_record
msg_type = _get_message_type(connection_info)
File "<stdin>", line 2, in tmp
File "/opt/ros/melodic/lib/python2.7/dist-packages/genpy/dynamic.py", line 138, in generate_dynamic
specs = {core_type: genmsg.msg_loader.load_msg_from_string(msg_context, core_msg, core_type)}
File "/opt/ros/melodic/lib/python2.7/dist-packages/genmsg/msg_loader.py", line 269, in load_msg_from_string
spec = MsgSpec(types, names, constants, text, full_name, package_name)
File "/opt/ros/melodic/lib/python2.7/dist-packages/genmsg/msgs.py", line 251, in __init__
raise InvalidMsgSpec("Duplicate field names in message: %s"%names)
genmsg.base.InvalidMsgSpec: Duplicate field names in message: ['header', 'seq', 'stamp', 'frame_id', 'hypotheses', 'id', 'active', 'merged_to', 'position', 'x', 'y', 'z', 'covariance', 'objectness', 'confirmed', 'class_probabilities', 'n_detections', 'last_detection_stamp', 'image', 'header', 'seq', 'stamp', 'frame_id', 'format', 'data']
>>>
And the message is:
$ rosmsg show darpa_msgs/LocalizationHypotheses
std_msgs/Header header
uint32 seq
time stamp
string frame_id
darpa_msgs/LocalizationHypothesis[] hypotheses
int32 id
bool active
int32 merged_to
geometry_msgs/Point position
float64 x
float64 y
float64 z
float64[9] covariance
float64 objectness
bool confirmed
float32[] class_probabilities
int32 n_detections
time last_detection_stamp
sensor_msgs/CompressedImage image
std_msgs/Header header
uint32 seq
time stamp
string frame_id
string format
uint8[] data
Is it possible that all the names that are being checked are not "namespaced" by their relative path in the message definition? I.e., putting two Header fields in different parts of the message is treated invalid?
The exception is raised here
https://github.com/ros/genmsg/blob/7d8b6ce6f43b6e39ea8261125d270f2d3062356f/src/genmsg/msgs.py#L250-L251
And the names reported by the exception contains the "non-namespaced" field names: ['header', 'seq', 'stamp', 'frame_id', 'hypotheses', 'id', 'active', 'merged_to', 'position', 'x', 'y', 'z', 'covariance', 'objectness', 'confirmed', 'class_probabilities', 'n_detections', 'last_detection_stamp', 'image', 'header', 'seq', 'stamp', 'frame_id', 'format', 'data']
So it seems that rosbag expects another format of the message definition - the one with ===-divided sections specifying each subtype separately, instead of this "all-in" message definition I showed in the issue description.
I just can't figure out how did this kind of definition get into our bags. I checked the bag directly via a debugger and the definition in the connection info is the "all-in" type. When I tried to recreate this situation artificially without running the whole pipeline, I tried roscpp publishers, rospy publishers, topic_tools relay, and feeding all of them to rosbag record (which is how we record the bags), but all of these produce the ===-sectioned definitions. The rosbag record normally runs in a container that doesn't know this message definition, and I even simulated that by calling record from a workspace which doesn't source the message definition package, but it still recorded the ===-separated definition (probably provided by publisher). Any ideas how could the "all-in" definitions sneak into the bags? We only use rospy and roscpp.
Even rosbag check is confused:
$ rosbag check /media/data/bags/subt_virtual/robot_data_0_TEAMBASE.bag
WARNING: For type [darpa_msgs/LocalizationHypotheses] stored md5sum [7da59d174d4c697d0571b3a6fc9fed73] has invalid message definition."
The following migrations need to occur:
* From: darpa_msgs/LocalizationHypotheses [d41d8cd98f00b204e9800998ecf8427e]
To: darpa_msgs/LocalizationHypotheses [7da59d174d4c697d0571b3a6fc9fed73]
1 rules missing:
* From: darpa_msgs/LocalizationHypotheses [d41d8cd98f00b204e9800998ecf8427e]
To: darpa_msgs/LocalizationHypotheses [7da59d174d4c697d0571b3a6fc9fed73]
But:
$ rosbag info /media/data/bags/subt_virtual/robot_data_0_TEAMBASE.bag
path: /media/data/bags/subt_virtual/robot_data_0_TEAMBASE.bag
version: 2.0
duration: 1hr 0:23s (3623s)
start: Jan 01 1970 01:00:48.84 (48.84)
end: Jan 01 1970 02:01:11.85 (3671.85)
size: 223.4 MB
messages: 1083074
compression: none [281/281 chunks]
types: darpa_msgs/LocalizationHypotheses [7da59d174d4c697d0571b3a6fc9fed73]
and
$ rosmsg md5 darpa_msgs/LocalizationHypotheses
7da59d174d4c697d0571b3a6fc9fed73
So rosbag check "invented" md5 d41d8cd98f00b204e9800998ecf8427e that doesn't correspond to anything in the bag or in the system.