Unity-Technologies
Reorder RayPerceptionSensor output to reflect its spatial structure
Is your feature request related to a problem? Please describe. When I'm processing the outputs of RayPerceptionSensorComponent3D, the rays seem to ordered in a spiral-like order -- first we get the output of the central ray, then the first on the left (?), then the first on the right, then the second on the left, etc. This is somewhat unintuitive, since we have a very clear spatial structure through the ordering of the rays, so we could, in principle, process them with a 1D convolution in the neural network. However, this is impossible without manually reordering the observations on the Python side, which in turn is a pain because we have to know all the parameters of the sensor.
Describe the solution you'd like The basic solution I'd like to see is simply reordering the rays so that the first entry is the left-most ray, and the last one is the right-most ray. I'm about 80% sure this would be as simple as modifying RayPerceptionComponentBase.cs#L209.
Describe alternatives you've considered A more "advanced" version of modifying this sensor that I'd be happy to see is giving more structure to the sensor and to the framestacking. If we're doing a 4-frame stack, with 15 rays, and 1 tag to detect, the resultant observation is a flat array of size 415(1+2) = 180. Instead, outputting a 3-dimensional tensor [4, 15, 3] would be much more informative - and it can be reduced to the former representation via a simple flattening. But this would probably be a bit more tricky, and to some extent this reshaping can also be done on the python side.
I don't know what the current policy on outside contributions is, but I'd be happy to contribute a PR for the ray reordering.
Thank you. We've logged this internally as https://jira.unity3d.com/browse/MLA-2537. Feel free to submit a PR. Please have a look at our contribution guidelines: https://github.com/Unity-Technologies/ml-agents/blob/main/com.unity.ml-agents/CONTRIBUTING.md
