Cannot set intrinsics with offset principal point for visualisation

[reubendocea]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for main branch).

Describe the issue

I am attempting to do the following:

1. Create a triangle mesh from an RGBD image, such that the vertices of the mesh correspond to the pixels of the RGBD after conversion to point clouds. The vertices are assigned colours according to the RGB of the original pixel. (Successful)
2. Render the mesh in Open3D visualisation using the original camera intrinsics which are used to derive the point cloud, such that the resulting render looks (almost) identical to the original RGB image. (Fails)

I have not been able to do this because Open3D will not allow me to set the principal point of the camera intrinsics.

Steps to reproduce the bug

def visualise(mesh, rgb, camera_parameters):
    # Set up Open3D visualization
    width, height = rgb.shape[1], rgb.shape[0]
    print("Visualising...")
    vis = o3d.visualization.Visualizer()
    vis.create_window(width=width, height=height)
    vis.add_geometry(mesh)

    # Get the view control and set camera parameters manually
    ctr = vis.get_view_control()
    camera_params = ctr.convert_to_pinhole_camera_parameters()

    # Set camera intrinsic parameters
    intrinsic = camera_params.intrinsic
    intrinsic.set_intrinsics(
        width=width,
        height=height,
        fx=camera_parameters['fx'],
        fy=camera_parameters['fy'],
        cx=camera_parameters['cx'],
        cy=camera_parameters['cy']
    )
    camera_params.intrinsic = intrinsic

    # Set camera extrinsic parameters (position at (0, 0, 0))
    extrinsic = np.identity(4)
    extrinsic[1, 1] = -1
    extrinsic[2, 2] = -1
    camera_params.extrinsic = extrinsic

    print("camera_params intrinsic BEFORE: ", camera_params.intrinsic)
    print("camera_params intrinsic_matrix BEFORE: ", camera_params.intrinsic.intrinsic_matrix)

    ctr.convert_from_pinhole_camera_parameters(camera_params, allow_arbitrary=True)

    vis.run()

    camera_params = ctr.convert_to_pinhole_camera_parameters()
    print("camera_params intrinsic AFTER: ", camera_params.intrinsic)
    print("camera_params intrinsic_matrix AFTER: ", camera_params.intrinsic.intrinsic_matrix)

    vis.destroy_window()

Error message

Here is the output with allow_arbitrary=True:

camera_params intrinsic BEFORE:  PinholeCameraIntrinsic with width = 534 and height = 371.
Access intrinsics with intrinsic_matrix.
camera_params intrinsic_matrix BEFORE:  [[534.195   0.    246.45 ]
 [  0.    534.095 181.37 ]
 [  0.      0.      1.   ]]
camera_params intrinsic AFTER:  PinholeCameraIntrinsic with width = 534 and height = 371.
Access intrinsics with intrinsic_matrix.
camera_params intrinsic_matrix AFTER:  [[534.095   0.    266.5  ]
 [  0.    534.095 185.   ]
 [  0.      0.      1.   ]]

The principal point does not remain as I set it. Nor do the fx and fy parameters.

Here is the output with allow_arbitrary=False:

camera_params intrinsic BEFORE:  PinholeCameraIntrinsic with width = 534 and height = 371.
Access intrinsics with intrinsic_matrix.
camera_params intrinsic_matrix BEFORE:  [[534.195   0.    246.45 ]
 [  0.    534.095 181.37 ]
 [  0.      0.      1.   ]]
[Open3D WARNING] [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
camera_params intrinsic AFTER:  PinholeCameraIntrinsic with width = 534 and height = 371.
Access intrinsics with intrinsic_matrix.
camera_params intrinsic_matrix AFTER:  [[321.2954248   0.        266.5      ]
 [  0.        321.2954248 185.       ]
 [  0.          0.          1.       ]]

Again, the fx and fy parameters do not remain as set, and the principal points are changed.

Expected behavior

My expectation, at least for allow_arbitraty=True, would be that all intrinsics parameters remain as I set them.

Open3D, Python and System information

- Operating system: Ubuntu 20.04
- Python version: Python 3.11
- Open3D version: output from python: 0.18.0+594f820 (I have tried all combinations of Python 3.8, 3.10, 3.11; together with open3d 0.16 and 0.18)
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip

Additional information

No response

[zerolover]

Yesterday I encountered the same problem but I don't know why until I found this issue. source code here The output is :

camera_params intrinsic_matrix BEFORE: 
 [[3605.15789     0.        947.328452]
 [   0.       3659.13751   523.233672]
 [   0.          0.          1.      ]]
camera_params intrinsic AFTER: 
 PinholeCameraIntrinsic with width = 1920 and height = 1080.
Access intrinsics with intrinsic_matrix.
camera_params intrinsic_matrix AFTER: 
 [[3659.13751    0.       959.5    ]
 [   0.      3659.13751  539.5    ]
 [   0.         0.         1.     ]]

[reubendocea]

The source code seems to go much deeper than that :fearful:

In the meantime, I found that trimesh and pyrender together serve my requirements :smile:

[zerolover]

The source code seems to go much deeper than that 😨

In the meantime, I found that trimesh and pyrender together serve my requirements 😄

Bingo! I use pyrender too. But pyrender seems a little slower than open3d, and pyrender use OpenGL camera coordinate

[brandonrwin]

Rearranging some calls can get around this: https://github.com/isl-org/Open3D/issues/1483#issuecomment-1873081439