MIT-SPARK
Data Provider - No Input received when I am trying Kimera with D435i
Dear All
Because I have to integrate with other applications running on Kinetic, I successfully compiled Kimera on Kinetic but (always a but :) ) I am facing different issues:
- When running the Euroc V1_01_easy bag, Kimera is waiting for a ground truth initialization but when I looked at the launch file, looks like it is not requested
fabrice@fabrice-desktop:~/catkin_kimera_ws$ roslaunch kimera_vio_ros kimera_vio_ros_euroc.launch
... logging to /home/fabrice/.ros/log/b9d34818-f346-11ea-8b5b-88b1116f3669/roslaunch-fabrice-desktop-11000.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://192.168.1.116:32873/
SUMMARY
========
PARAMETERS
* /kimera_vio_ros/kimera_vio_ros_node/base_link_frame_id: base_link
* /kimera_vio_ros/kimera_vio_ros_node/left_cam_frame_id: cam0
* /kimera_vio_ros/kimera_vio_ros_node/map_frame_id: map
* /kimera_vio_ros/kimera_vio_ros_node/mono_ransac_threshold: 30
* /kimera_vio_ros/kimera_vio_ros_node/online_run: True
* /kimera_vio_ros/kimera_vio_ros_node/params_folder_path: /home/fabrice/cat...
* /kimera_vio_ros/kimera_vio_ros_node/position_det_threshold: 0.3
* /kimera_vio_ros/kimera_vio_ros_node/right_cam_frame_id: cam1
* /kimera_vio_ros/kimera_vio_ros_node/stereo_ransac_threshold: 20
* /kimera_vio_ros/kimera_vio_ros_node/use_lcd: False
* /kimera_vio_ros/kimera_vio_ros_node/use_online_cam_params: False
* /kimera_vio_ros/kimera_vio_ros_node/velocity_det_threshold: 0.1
* /kimera_vio_ros/kimera_vio_ros_node/world_frame_id: world
* /kimera_vio_ros/posegraph_viewer/frame_id: world
* /rosdistro: kinetic
* /rosversion: 1.12.14
* /use_sim_time: True
NODES
/kimera_vio_ros/
kimera_vio_ros_node (kimera_vio_ros/kimera_vio_ros_node)
posegraph_viewer (pose_graph_tools/visualizer_node)
auto-starting new master
process[master]: started with pid [11010]
ROS_MASTER_URI=http://192.168.1.116:11311
setting /run_id to b9d34818-f346-11ea-8b5b-88b1116f3669
process[rosout-1]: started with pid [11023]
started core service [/rosout]
process[kimera_vio_ros/kimera_vio_ros_node-2]: started with pid [11041]
process[kimera_vio_ros/posegraph_viewer-3]: started with pid [11048]
[ INFO] [1599729568.694822036]: Initializing pose graph visualizer
I0910 11:19:28.779866 11041 RegularVioBackEnd.cpp:114] Using Regular VIO backend.
I0910 11:19:28.780025 11041 PipelineModule.h:449] MISO Pipeline Module: Display has no output queue registered.
I0910 11:19:28.780292 11041 Pipeline.cpp:495] Pipeline Modules launched (parallel_run set to 1).
I0910 11:19:28.781617 11041 RosOnlineDataProvider.cpp:45] Waiting for ROS time to be valid...
(Sim Time is enabled; run rosbag with --clock argument)
I0910 11:19:36.233929 11041 RosOnlineDataProvider.cpp:56] Requested initialization from ground-truth. Initializing ground-truth odometry one-shot subscriber.
W0910 11:19:36.236655 11041 RosOnlineDataProvider.cpp:65] Waiting for ground-truth pose to initialize VIO on ros topic: /kimera_vio_ros/gt_odom
E0910 11:19:39.343837 11041 RosOnlineDataProvider.cpp:82] Missing ground-truth pose while trying for 3.0099 seconds.
Enabling autoInitialize and continuing without ground-truth pose.
I0910 11:19:39.369618 11203 Pipeline.h:182] Spinning Kimera-VIO.
I0910 11:19:39.369642 11041 KimeraVioRos.cpp:114] Statistics
----------- # Log Hz {avg +- std } [min,max]
Display [ms] 0
Mesher [ms] 0
VioBackEnd [ms] 0
VioFrontEnd [ms] 0
Visualizer [ms] 0
backend_input_queue Size [#] 0
data_provider_left_frame_queue Size [#] 0
data_provider_right_frame_queue Size [#] 0
display_input_queue Size [#] 0
mesher_backend Size [#] 0
mesher_frontend Size [#] 0
stereo_frontend_input_queue Size [#] 0
visualizer_backend_queue Size [#] 0
visualizer_frontend_queue Size [#] 0
visualizer_mesher_queue Size [#] 0
W0910 11:19:39.621721 11203 DataProviderModule.cpp:103] Waiting for IMU data...
W0910 11:19:39.670670 11203 DataProviderModule.cpp:103] Waiting for IMU data...
- I also tried to connect Kimera with a D435i following your tutorial: https://github.com/MIT-SPARK/Kimera-VIO-ROS/blob/master/docs/hardware_setup.md
Here, I scratched my head, because Kimera does not received any data. When I display the nodes and topics I just see only /camera/imu as a topic to which Kimera subscribes. I used image_view to check that there are images, camera_info, everything is ok.
I0910 15:51:20.510844 21504 RosDataProviderInterface.cpp:111] ========================================
I0910 15:51:20.534885 21504 KimeraVioRos.cpp:88] Connecting Vio Pipeline and Data Provider.
I0910 15:51:20.535038 21504 KimeraVioRos.cpp:114] Statistics
----------- # Log Hz {avg +- std } [min,max]
Mesher [ms] 0
VioBackEnd [ms] 0
VioFrontEnd [ms] 0
Visualizer [ms] 0
backend_input_queue Size [#] 0
data_provider_left_frame_queue Size [#] 0
data_provider_right_frame_queue Size [#] 0
display_input_queue Size [#] 0
mesher_backend Size [#] 0
mesher_frontend Size [#] 0
stereo_frontend_input_queue Size [#] 0
visualizer_backend_queue Size [#] 0
visualizer_frontend_queue Size [#] 0
visualizer_mesher_queue Size [#] 0
I0910 15:51:20.536837 21619 PipelineModule.h:190] Module: Display - Spinning.
I0910 15:51:20.536845 21620 Pipeline.h:182] Spinning Kimera-VIO.
I0910 15:51:20.536877 21620 PipelineModule.h:190] Module: Data Provider - Spinning.
I0910 15:51:20.670258 21620 DataProviderModule.cpp:77] Skipping first frame, because we do not have a concept of a previous frame timestamp otherwise.
W0910 15:51:20.670294 21620 PipelineModule.h:217] Module: Data Provider - No Input received.
W0910 15:51:20.728554 21620 PipelineModule.h:217] Module: Data Provider - No Input received.
I0910 15:51:20.738580 21573 FeatureDetector.cpp:185] Number of points detected : 800
I0910 15:51:20.738612 21573 FeatureDetector.cpp:197] Need n corners: 800
I0910 15:51:20.738687 21573 NonMaximumSuppression.cpp:87] Running RangeTree: 4
I0910 15:51:20.740181 21573 NonMaximumSuppression.cpp:105] Non Maximum Suppression Timing [ms]: 1
W0910 15:51:20.769744 21620 PipelineModule.h:217] Module: Data Provider - No Input received.
I0910 15:51:20.801784 21620 ThreadsafeQueue.h:308] Queue with id: stereo_frontend_input_queue is getting full, size: 2
W0910 15:51:20.801831 21620 PipelineModule.h:217] Module: Data Provider - No Input received.
I0910 15:51:20.802299 21573 FeatureDetector.cpp:241] Corner Sub Pixel Refinement Timing [ms]: 62
^CI0910 15:51:20.837973 21620 ThreadsafeQueue.h:308] Queue with id: stereo_frontend_input_queue is getting full, size: 3
W0910 15:51:20.838016 21620 PipelineModule.h:217] Module: Data Provider - No Input received.
I0910 15:51:20.850323 21574 VioBackEnd.h:175] ------------------- Initialize Pipeline: timestamp = 1599745880704129934--------------------
I0910 15:51:20.850323 21573 StereoVisionFrontEnd.cpp:128] ------------------- Processing frame k = 2--------------------
W0910 15:51:20.850354 21574 InitializationFromImu.cpp:25] InitializationFromImu: assumes that the vehicle is stationary and upright along some axis,and gravity vector is along a single axis!
I0910 15:51:20.850457 21573 StereoVisionFrontEnd.cpp:153] Current IMU Preintegration frequency: 116279 Hz. (86 us).
I0910 15:51:20.852085 21574 VioBackEnd.cpp:1183] Backend: Update IMU Bias.
W0910 15:51:20.852221 21575 Mesher.cpp:1380] Missing landmark information to build 3D Mesh.
I0910 15:51:20.854032 21575 Mesher.cpp:1701] Delaunay Triangle out of image (size: x: 0, y: 0, height: 480, width 640
Triangle: x, y:
1920, 0
0, 1920
624, 211
Finally I have a practical question concerning IMU calibration of the D435i? It appears that the calibration is not so good, specially regarding the linear acceleration, like x or y axis = 0.5 m/s/s when the camera is fixed on a tripod and no movement at all - how do you manage this in Kimera?
Thank you very much for your support
Please give also the following information:
- SparkVio branch, tag or commit used: V4.0
- GTSAM version used: 4.1rc
- OpenGV version used:
- OpenCV version used: type
opencv_version - Operating system and version (e.g. Ubuntu 16.04 or Windows 10): Ubuntu 16.04 + ROS kinetic
- Did you change the source code? (yes / no): No
I used Kalibr to calibrate the camera and this time I got the following message:
I0928 09:33:04.050410 5641 KimeraVioRos.cpp:58] Destroy Ros Display.
I0928 09:33:04.050428 5641 KimeraVioRos.cpp:62] Creating Ros Display.
I0928 09:33:04.092790 5641 KimeraVioRos.cpp:67] Destroy Vio Pipeline.
I0928 09:33:04.092811 5641 KimeraVioRos.cpp:71] Creating Kimera-VIO.
F0928 09:33:04.096122 5641 StereoCamera.cpp:72] Check failed: baseline_ > 0.0 (-0.0502485 vs. 0)
but when I look at the code:
// Calc baseline (see L.2700 and L.2616 in
// https://github.com/opencv/opencv/blob/master/modules/calib3d/src/calibration.cpp
// NOTE: OpenCV pose convention is the opposite, therefore the missing -1.0
CHECK_NE(Q_.at<double>(3, 2), 0.0);
baseline_ = 1.0 / Q_.at<double>(3, 2);
CHECK_GT(baseline_, 0.0);
I am wondering how is it possible to get 0 by dividing by Q[3,2] ?
Hi @FaboNo, the error isn't really about getting a 0, right? It is about the baseline_ being negative (-0.0502485).
The breaking check is the one in line 72 of StereoCamera.cpp.
Can you show me the {Left,Right}CameraParams.yaml that you are using and the Kalibr pdfs?
@ToniRV Thank you for your message
Actually I tried many calibrations and I used this one those values are quite close to the values provided by infra1/infra2/camera_info (I assume that they are reliable):
cam0:
cam_overlaps: [1]
camera_model: pinhole
distortion_coeffs: [0.327534603927029, 0.21450632361513983, -0.11299645863294606,
0.1538427582357007]
distortion_model: equidistant
intrinsics: [382.47776119570256, 382.6312369917512, 320.8597427296652, 237.53543049146265]
resolution: [640, 480]
rostopic: /camera/infra1/image_rect_raw
cam1:
T_cn_cnm1:
- [0.9999995987711771, -3.056367043916324e-06, 0.0008957946985143786, -0.05017249832103017]
- [3.1189758319643222e-06, 0.9999999975527939, -6.989050527803715e-05, 0.00010651164702781861]
- [-0.0008957944827111481, 6.989327119796712e-05, 0.9999995963335054, 0.00023235999387130802]
- [0.0, 0.0, 0.0, 1.0]
cam_overlaps: [0]
camera_model: pinhole
distortion_coeffs: [0.3130360953854527, 0.32548996775879063, -0.4388114751551025,
0.47271139692383496]
distortion_model: equidistant
intrinsics: [383.08396077104817, 383.2196530776498, 320.79573033200455, 237.44391499975424]
resolution: [640, 480]
rostopic: /camera/infra2/image_rect_raw
Here is the pdf edited by Kalibr report-cam-homefabrice2020-09-25-16-45-16.pdf
I saw that the reprojection errors are not so good - I heard that for a good calibration, one should get a reprojection error below 0.5pix
I did not upload the cameratoimu calibration because it is too big, but the reprojection errors increased a bit (more than 1 pixel), probably due to the IMU noise - did you observe the same thing on your side?
Here is the result of Kalib_all to compute both the noise density and random walk for the gyroscope and the accelerometer (https://github.com/rpng/kalibr_allan)
Do you think that these values are reasonable (based on 2 hours IMU recording - camera at rest).
I realize that Visual Inertial Odometry(I still work on the math as well) is very tough to tune specially with the D435i which is equipped with a "poor" IMU. Large drift when the camera is moving... from your experience is it due to bad camera calibration or mainly IMU noise?
Thank you for this great work