OAK-FFC-4P StereoDepth set depth alignment with RGB Camera

[michaelnguyen11]

Describe the bug In our OAK-FFC-4P setup, the CAM_B is left, CAM_C is right and CAM_D is center. However, when configure StereoDepth to set depth alignment with RGB camera (center) stereo.setDepthAlign(dai.CameraBoardSocket.CAM_D), it has error :

[14442C1081F4D5D600] [1.2.1] [2.628] [StereoDepth(3)] [error] DepthAlign is only implemented for RGB camera and LEFT and RIGHT stereo inputs!

Change to stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A), look like it's success :

[14442C1081F4D5D600] [1.2.1] [2.615] [StereoDepth(3)] [warning] Align to RGB requested, but ColorCamera not configured. Can't determine output resolution automatically. Using MonoCamera resolution. If you want to set explicitly use 'setOutputSize' API.
[14442C1081F4D5D600] [1.2.1] [2.622] [StereoDepth(3)] [error] RGB camera calibration missing, aligning to RGB won't work

Minimal Reproducible Example

import numpy as np
import cv2
import depthai as dai

pipeline = dai.Pipeline()

cam_center = pipeline.create(dai.node.ColorCamera)
cam_center.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1200_P)
cam_center.setIspScale(400, 1200)
cam_center.setFps(30)
cam_center.setBoardSocket(dai.CameraBoardSocket.CAM_D)
# For now, RGB needs fixed focus to properly align with depth.
# This value was used during calibration
try:
    calibData = dai.Device().readCalibration2()
    lensPosition = calibData.getLensPosition(dai.CameraBoardSocket.CAM_D)
    if lensPosition:
        cam_center.initialControl.setManualFocus(lensPosition)
except:
    raise

cam_left = pipeline.create(dai.node.ColorCamera)
cam_left.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1200_P)
cam_left.setIspScale(400, 1200)
cam_left.setFps(30)
cam_left.setBoardSocket(dai.CameraBoardSocket.CAM_B)

cam_right = pipeline.create(dai.node.ColorCamera)
cam_right.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1200_P)
cam_right.setIspScale(400, 1200)
cam_right.setFps(30)
cam_right.setBoardSocket(dai.CameraBoardSocket.CAM_C)

stereo = pipeline.create(dai.node.StereoDepth)
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.setLeftRightCheck(True)
stereo.setExtendedDisparity(False)
stereo.setSubpixel(False)
# stereo.setDepthAlign(dai.CameraBoardSocket.CAM_D)

# Define streams
rgbOut = pipeline.create(dai.node.XLinkOut)
rgbOut.setStreamName("center")
disparityOut = pipeline.create(dai.node.XLinkOut)
disparityOut.setStreamName("disparity")

# Linking
cam_center.isp.link(rgbOut.input)
cam_left.isp.link(stereo.left)
cam_right.isp.link(stereo.right)
stereo.disparity.link(disparityOut.input)

with dai.Device() as device:
    device.startPipeline(pipeline)

    frameRgb = None
    frameDisp = None

    centerQueue = device.getOutputQueue(name='center', maxSize=4, blocking=False)
    dispQueue = device.getOutputQueue(name='disparity', maxSize=4, blocking=False)

    while True:
        frameRgb = centerQueue.get().getCvFrame()
        frameDisp = dispQueue.get().getFrame()
        maxDisparity = stereo.initialConfig.getMaxDisparity()
        frameDisp = (frameDisp * 255. / maxDisparity).astype(np.uint8)

        frameDisp = cv2.applyColorMap(frameDisp, cv2.COLORMAP_TURBO)
        frameDisp = np.ascontiguousarray(frameDisp)

        # Need to have both frames in BGR format before blending
        if len(frameDisp.shape) < 3:
            frameDisp = cv2.cvtColor(frameDisp, cv2.COLOR_GRAY2BGR)

        blended = cv2.addWeighted(frameRgb, 0.4, frameDisp, 0.6, 0)

        cv2.imshow('rgb + depth', blended)
        frameRgb = None
        frameDisp = None

        if cv2.waitKey(1) == ord('q'):
            break

cv2.destroyAllWindows()

Expected behavior How to overcome this issue if I sill want to keep the CameraBoardSocket CAM_D connect to center sensor.

Attach system log

• If I set stereo.setDepthAlign(dai.CameraBoardSocket.CAM_D), the error logs are:

[14442C1081F4D5D600] [1.2.1] [2.628] [StereoDepth(3)] [error] DepthAlign is only implemented for RGB camera and LEFT and RIGHT stereo inputs!

• If I set stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A), the error logs are:

[14442C1081F4D5D600] [1.2.1] [2.615] [StereoDepth(3)] [warning] Align to RGB requested, but ColorCamera not configured. Can't determine output resolution automatically. Using MonoCamera resolution. If you want to set explicitly use 'setOutputSize' API.
[14442C1081F4D5D600] [1.2.1] [2.622] [StereoDepth(3)] [error] RGB camera calibration missing, aligning to RGB won't work

[jakaskerl]

Hi @michaelnguyen11
Could you paste the camera calibration you have done for your current setup?

[michaelnguyen11]

Hi @jakaskerl ,

Please take a look at attached file for the calibration file (generated from calibration_reader.py script).

calib_14442C1081F4D5D600.zip