Correct commands to get the OpticalFlow

[COATZ]

Question

What's your question?

Hi,

Thanks a lot for adding the OpticalFlow feature to AirSim but I am having some issue getting it.

What are the correct options of the airsim.ImageRequest to get the 2-channels ground truth OpticalFlow from my front drone camera (not the OpticalFlowVis) using the Python API.

My camera parameters are width = 100, height = 100 and I am currently using: responses = self.drone.simGetImages([airsim.ImageRequest("front", 8, True, False)]). But the output is a single channel image instead of the two expected: print(len(responses[0].image_data_float)) > 10000 (and not 20000 as expected)

I tried with the option "pixel_as_float=False" and got a 3-channels output of np.uint8. But the raw ground truth optical flow should be float so I am guessing that is not the right option...

Thank you!

[zimmy87]

Since simGetImages just returns the buffer of a SceneCaptureComponent2D instance, I would not expect to get output with multiple entries for each channel. Looking at how OpticalFlow's post process material is defined here, the 2 channels (vx & vy) are mapped to the emissive color of the material, meaning that these channels will be encoded in the pixel color of the output of simGetImages. I don't know exactly how Unreal's material editor maps a float2 to a color value, so you may need to play around with the output, but it looks like you need to parse each pixel's color for the values of both channels. If you find out how this conversion works, please post that in this issue.

[BBBBBBob]

Hi, I have met the same problem, have you found a solution to fix it?

[BBBBBBob]

In the material, x-velocity is encoded in the red channel and y-velocity in the green channel. If we observe the optical flow image, there are only red and green colors in the OpticalFlow setting. However, Unreal Engine only returns the black color to the material when the speed is non-positive. It is correct that the optical flow image is dark when the vehicle is at a standstill, but it does not return any optical flow when the camera moves to the right side or upwards. Here is an example: This problem only exists in the OpticalFlow setting.

[wangpinzhi]

Hello, have you found a solution? I met the same problem.