probabilists
Requested flow architectures
Description
This issue tracks the flow/transformation architectures that are requested and/or implemented. You are welcome to request new architectures. If you wish to contribute to Zuko, this list is the perfect place to start.
Typically, implementing a new flow architecture consists in adding a new transformation in zuko/transforms.py and a new class in the zuko/flows/ directory. NSF and SOSPF are great examples. Sometimes, a special LazyTransform needs to be implemented to take into account the specificities of the architecture. NAF and GF are such examples.
List of requested flows
- [x] NICE by Dinh et al. (2014)
- [ ] Planar flow by Rezende et al. (2015)
- [ ] Radial flow by Rezende et al. (2015)
- [ ] Multi-scale flow(s) such as RealNVP by Dinh et al. (2016) or Glow by Kingma et al. (2018)
- [x] Masked autoregressive flow by Papamakarios et al. (2017)
- [ ] Sylvester normalizing flows by van den Berg et al. (2018)
- [x] Neural autoregressive flow by Huang et al. (2018)
- [x] Continuous flow by Chen et al. (2018)
- [x] Free-form continuous flow by Grathwohl et al. (2018)
- [x] Sum-of-squares polynomial flow by Jaini et al. (2019)
- [ ] Residual flow by Chen et al. (2019)
- [x] Neural spline flow by Durkan et al. (2019)
- [x] Unconstrained monotonic neural flow by Wehenkel et al. (2019)
- [x] Circular spline flow by Rezende et al. (2020)
- [ ] Augmented flow(s) by Huang et al. (2020)
- [ ] Rational-linear spline flow by Dolatabadi et al. (2020)
- [x] Gaussianization flow by Meng et al. (2020)
Hello François, thank you for developing, releasing and maintaining zuko, it's an incredible package. Two missing flows that could be added to the list are
- Sylvester Flows van den Berg et al. (2019)
- LU-Net Chan et al. (2022)
Hi @bhvieira, thank you for the kind words and suggestions! I have added the Sylvester flow to the list. I believe the idea of using LU-decomposition for invertible linear layers comes from Kingma et al. (2018) and later re-used by Durkan et al. (2019). LULinearTransform is already implemented in Zuko actually. However, it could likely be improved by using torch.linalg.lu_factor and torch.linalg.lu_solve.
LU-Net does not actually solve the LU decomposition, it's just based on composing two linear layers: one lower triangular and the other upper triangular. The inverse is based on solving a system of linear equations.
Kingma et al. (2018) and Durkan et al. (2019) do exactly what you describe: they parameterize a linear transformation with triangular matrices $L$ and $U$ (and permutation $P$). They can inverse these transformations efficiently because $L$ and $U$ are triangular. The only time the LU-decomposition is performed is at the initialization of the network.