Sparse tensor and Sparse CudaTensor

[mratsim]

Sparse tensor support is important in general machine learning especially for to store matrices of one-hot encoded vectors.

For CUDA backend, CuBLAS has a Sparse API. For CPU, further investigation is needed to find a suitable Sparse BLAS backend. See this link for potential libraries.

[mratsim]

Progress on the research:

The main challenge is finding an up-to-date Sparse BLAS library.

Survey of the field:

• http://www.netlib.org/utk/people/JackDongarra/la-sw.html

Updated:

• MKL
• Scipy but not parallel.

Seems state of the art:

• ViennaCL via their OpenMP backend is much! faster than MKL, see SpMM benchmark. Work on CSR matrices. C++.
• librsb and Julia wrapper. Custom Matrix Format. C/Fortran.

MPI / Distributed:

• CombBLAS

Full sparse tensor library:

• Amazon DSSTNE
• Netflix Vectorflow

Seems staled:

• PSBLAS (Parallel SparseBlas)
• Exascience
• Apple Accelerate (Accessible on iPhones)

Paper:

• Parallel SPMV using compressed Sparse Blocks

[mratsim]

"BlockSparse" optimized GPU kernels by OpenAI:

• Announcement
• Paper
• https://github.com/openai/blocksparse

[chenpeizhi]

The TACO tensor compiler generates efficient dense, sparse, and block sparse kernels for various formats. It is worth checking out.