Holes in depthmap

[DonMaEn]

Bug report

• AirSim Version/#commit:version 1.7.0 / commit: 0becf9148d88c19c3ffd69359ebe1f9c3ae43826
• UE/Unity version: N/A
• autopilot version: N/A
• OS Version: centos stream 8

What's the issue you encountered?

When using the orthographic camera, holes appear in the depth map images saved to disk when using the client.startRecording() function. These holes only appear on the ground mesh, not on buildings or trees. I have tested this using the pre-compiled AirSimNH scene and the LandscapeMountains scene. In the below images you can see in yellow the hole, and you can see how the trees are not affected by this bug. The RGB image is from the exact same area as the depthmap image, and you can see there is no visible hole in the ground mesh.

Also, see the following screen shot of a point cloud I created from the depth maps and RGB images that shows where the holes appear in the AirSimNH scene:

Settings

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision",
  "Recording": {
    "RecordOnMove": true,
    "RecordInterval": 0.05,
    "Cameras": [
        { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "Compress": true },
        { "CameraName": "0", "ImageType": 5, "PixelsAsFloat": false, "Compress": true },
        { "CameraName": "0", "ImageType": 1, "PixelsAsFloat": true, "Compress": false }
    ]
  },
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 728,
        "Height": 728,
        "OrthoWidth": 60,
        "AutoExposureSpeed": 100,
        "MotionBlurAmount": 0,
        "ProjectionMode": "orthographic"
      },
	  {
        "ImageType": 1,
        "Width": 728,
        "Height": 728,
        "OrthoWidth": 60,
        "AutoExposureSpeed": 100,
        "MotionBlurAmount": 0,
        "ProjectionMode": "orthographic"
      },
      {
        "ImageType": 5,
        "Width": 728,
        "Height": 728,
        "OrthoWidth": 60,
        "AutoExposureSpeed": 100,
        "MotionBlurAmount": 0,
        "ProjectionMode": "orthographic"
      }
    ]
  },
  "SubWindows": [
    {"WindowID": 0, "CameraName": "0", "ImageType": 3, "VehicleName": "", "Visible": true},
    {"WindowID": 1, "CameraName": "0", "ImageType": 5, "VehicleName": "", "Visible": true},
    {"WindowID": 2, "CameraName": "0", "ImageType": 0, "VehicleName": "", "Visible": true}
  ],
  "SegmentationSettings": {
    "InitMethod": "",
    "MeshNamingMethod": "",
    "OverrideExisting": false
  }
}

How can the issue be reproduced?

1. Use the pre-compiled AirSimNH scene and the above settings file
2. Collect images using the airsim record feature in a lawnmower style pattern, similar to the following script: https://github.com/MizzouINDFUL/UEUAVSim/blob/main/airsim/airsim_collect.py
3. Collect the images with pitch = -90, i.e. directly downward

Include full error message in text form

N/A

Notes:

Because I am using centos stream 8 I have not been able to compile Airsim from source, and therefore could not test on the master branch.

I tried using the latest version 1.8 pre-compiled binaries, however when running my collection script airsim crashes. I'm guessing this is because I am still using version 1.7 of the airsim python API. 1.8 is not yet available on pypi so I cannot test this theory.

[jonyMarino]

Thanks for the report @DonMaEn! The holes always appear in the same places or this is a stochastic behavior?

[DonMaEn]

@jonyMarino, good question. The holes do not always appear in the same place. The following image is a point cloud I created using the same method, but of a smaller area. You can still see the holes, but they are in different locations.

[artths]

In my case, holes also appear randomly on RGB camera images (Scene) in the form of randomly shaped white spots, on DepthPerspective it is a spot with a constant maximum value of 65k. DepthVis frame is completely black with the exception of random white spots. This mainly happens when the drone turns. I don't notice it in the simulator itself. I use the latest official build for Windows. I tried to run the simulator on both Nvidia and AMD GPUs.

[artths]

OK found a way around the problem. Repeating the same query produces normal images.