ConCat layer Issue

[pkdeep]

Hi @auphelia @williamliao28, I am facing this issue where Concat layer is not getting converted to HLS layer. I tried the transformation InferConCatLayer(), but still it dint infer streaming Concat layer. Can you help with this. I am attacing original and converted .onnx files

Here is the original post https://github.com/Xilinx/finn/issues/853#issuecomment-1640254839

[iksnagreb]

There are scalar multiplications of floating-point type preceding the inputs to the Concat operator, turning the inputs into floating-point as well. Currently, the StreamingConcat does not support floating-point inputs, see here: https://github.com/Xilinx/finn/blob/dev/src/finn/transformation/fpgadataflow/convert_to_hls_layers.py#L1659

Unfortunately, this seems not to be easy to streamline, as the scale factors on each input branch are different. Streamlining the Mul nodes past the Concat layer only works out if they are all equal... It could work using some group-wise scale-factors afterwards, but I am not sure whether there is support for this.

Allowing the StreamingConcat operation to handle floating-point inputs is also not really desirable, as it would leave floating-points in the middle of your model graph (with finn you ideally want to have a continuous chain of integer-only operations). While it should be technically possible, this would require some refactoring of the corresponding HLS code generation.

[pkdeep]

Hi @iksnagreb . Thanks for the reply. I could get the Mul factors to be equal. Can this be processed to have a streaming replacement?

I am attaching both the pre-dataflow creation (after HLS) model image and after dataflow creation image. Any insight would be of great help to convert this into a fully streamlined model.

[iksnagreb]

Hm, conceptually, you now would need something like the MoveIdenticalOpPastJoinOp transformation (this is in finn.transformation.streamline.reorder), but it seems to be made for two-input join-nodes, while you have an "arbitrary" many inputs join-node. I am not sure whether there already is a suitable streamlining transformation somewhere in finn, but adapting the existing MoveIdenticalOpPastJoinOp or introducing a new, similar one seems not too complicated.

[auphelia]

Closing this issue, thanks @iksnagreb for the suggestions. @pkdeep, if you have tried out to extend MoveIdenticalOpPastJoinOp and would like to contribute to the finn compiler, we're always welcoming contributions from the community 🙂 Please refer to the contribution guidelines in that case.