Orbbec Femto-bolt IMU-Camera parameter For Help

[hplegend]

Thanks for you great work. I want to know whether you sdk provide a interface to obtain IMU-Camera parameters with Orbbec Femto-bolt device?

[kaatrasa]

Hey,

On some devices (at least with Orbbec Femto Mega), you can get imu->depth_camera extrinsics directly from OrbbecSDK (imu->color_camera wasn't directly available when I last checked):

const OBCalibrationParam &calibration = obPipeline.getCalibrationParam(obPipeline.getConfig());
const auto &e = calibration.extrinsics[OB_STREAM_GYRO][OB_STREAM_DEPTH];
const auto &R = e.rot;
const auto &t = e.trans;

From that you can compute imu->color_camera extrinsics yourself (imu->color = imu->depth->color = depth->color * imu->depth)

On other devices (Astra 2), we had to do the camera/imu calibration ourselves. You can see what camera parameters our SDK is using, by recording a dataset using the sai-record-orbbec tool, and checking calibration.json. For Orbbec Femto Mega & Bolt, the imu->color_camera is:

"imuToCamera": [
    [
        -3.0371e-05,
        -1,
        -0.00034758,
        0.0381531
    ],
    [
        -0.00123998,
        -0.000347542,
        0.999999,
        0.0284239
    ],
    [
        -0.999999,
        3.0802e-05,
        -0.00123996,
        -0.0368053
    ],
    [
        0,
        0,
        0,
        1
    ]
]

[hplegend]

Hey,

On some devices (at least with Orbbec Femto Mega), you can get imu->depth_camera extrinsics directly from OrbbecSDK (imu->color_camera wasn't directly available when I last checked):

const OBCalibrationParam &calibration = obPipeline.getCalibrationParam(obPipeline.getConfig());
const auto &e = calibration.extrinsics[OB_STREAM_GYRO][OB_STREAM_DEPTH];
const auto &R = e.rot;
const auto &t = e.trans;

From that you can compute imu->color_camera extrinsics yourself (imu->color = imu->depth->color = depth->color * imu->depth)

On other devices (Astra 2), we had to do the camera/imu calibration ourselves. You can see what camera parameters our SDK is using, by recording a dataset using the sai-record-orbbec tool, and checking calibration.json. For Orbbec Femto Mega & Bolt, the imu->color_camera is:

"imuToCamera": [
    [
        -3.0371e-05,
        -1,
        -0.00034758,
        0.0381531
    ],
    [
        -0.00123998,
        -0.000347542,
        0.999999,
        0.0284239
    ],
    [
        -0.999999,
        3.0802e-05,
        -0.00123996,
        -0.0368053
    ],
    [
        0,
        0,
        0,
        1
    ]
]

Thanks for you immidiately reply. I have notice the parameter in 'calibration.json', and the "imuToCamera" parameter is calcualte realtime by the approach 'depth->color * imu->depth' as you provide or it was calibrated offline by Kalibr?

[kaatrasa]

For Orbbec devices, we are using hardcoded imu->camera extrinsics (instead of computing them in real-time). I also now recalled this issue: https://github.com/orbbec/OrbbecSDK/issues/52, so we are still using manually calibrated values also for Femto Mega/Bolt.

Some links:

1. Our calibration instructions
2. Overriding default imu->camera in our Orbbec wrapper