strayrobots
Consecutive depth images having significant differences
The depth map is captured at a rate of fps. Among these depth maps, if the device is static and environment is static, there shouldn't be any significant difference except for minor sensor noises. However, I am seeing that whenever there is some surface transition in the recording, the consecutive depth images have some significant differences, whereas their difference image should be all black almost. But when the surface transition in the recording is very less, then consecutive depth images are quite similar, although there are some differences in pixel values but the difference image is more or less black. Any idea why this is happening? And how to resolve?
I'm not sure what exactly you mean, but I think you mean there are some unexplainable artifacts appearing on some transitions.
It seems that there is some ML post-processing happening in the SDK from Apple. That is, we don't actually get the sparse time-of-flight output from the sensor. The depth map goes through some post processing where the time-of-flight sensor output goes through some further steps. Based on the quality of the output, it seems there is some ML processing happening, possibly informed by the RGB image to produce the depth map.
I'm not aware that there is a way to get the raw sensor output, but do let me know if there is, as that could be helpful in many applications.
Well let me make it more clear. Let's assume the device is static and the environment is static as well. Now, if I record for let's say 15s, I will generate nearly 450 depth images if fps = 30. Among these depth images, the consecutive depth images should be similar right? For ex, depth img 10 and 12 should be more similar than depth img 10 and 400. However, after plotting the pixel wise difference image, I am seeing consecutive depth images have significant differences in the region where surfaces change from one to another. Take a look at the attached image. Also, do you have any reference to arkit docs where it's mentioned that the depth images goes through some post processing before being accessed by the devs?
I haven't seen any details on how they are producing the depth images. This is just based on my experience and the artifacts that can sometimes be seen near thin structures. This leads me to believe there is some kind of post processing happening before handing the depth map through the api. That looks like it might be what you are observing here.