Support for sample-parallelism (MMV)

[fpjentzsch]

Overview

Introduces support for additional parallelism along the spatial feature map dimension, i.e., processing multiple 'pixels' or 'samples' simultaneously. For some layers (e.g., MVAU), this functionality already existed in the HLS back-end and is now integrated on the compiler side. For others (e.g., RTL SWG), back-end functionality is extended.

Currently, this parallelism is controlled by the M attribute and requires existing folding factors (SIMD, PE, parallel_window) to be maxed out first.

Components

• RTL ConvolutionInputGenerator (done)
  • M controls the number of parallel input samples and output windows, i.e., mmv_in=M*1 and mmv_out=M*k_h*k_w.
  • Limitations on output FM W dimension: out_dim_w divisible by M, out_dim_w / M > 2 (1D conv is normalized to H=1).
  • Input FM dimensions are padded internally.
• MVAU (done)
  • Converts to and from MultiChanData internally, so that external interface is still a hls::stream<ap_uint<>>.
• VVAU (done)
  • Converts to and from MultiChanData internally, so that external interface is still a hls::stream<ap_uint<>>.
  • Requires hlslib PR to restore MMV support (https://github.com/Xilinx/finn-hlslib/pull/123).
• Padding (done, except for some cleanup)
  • Implement in RTL variant
• Pooling (work in progress)
  • likely implement only in Pool_Batch, not StreamingMaxPool_Batch variant
• DataWidthConverter (work in progress)

Tests

Integration into existing unit tests:

• test_fpgadataflow_slidingwindow_rtl
• ...

New:

• test_convert_to_hls_conv_mmv (tests a graph of multiple layers including padding and pooling)