Requested flow architectures

[francois-rozet]

Description

This issue tracks the flow/transformation architectures that are requested and/or implemented. You are welcome to request new architectures. If you wish to contribute to Zuko, this list is the perfect place to start.

Typically, implementing a new flow architecture consists in adding a new transformation in zuko/transforms.py and a new class in the zuko/flows/ directory. NSF and SOSPF are great examples. Sometimes, a special LazyTransform needs to be implemented to take into account the specificities of the architecture. NAF and GF are such examples.

List of requested flows

• NICE by Dinh et al. (2014)
• Planar flow by Rezende et al. (2015)
• Radial flow by Rezende et al. (2015)
• Multi-scale flow(s) such as RealNVP by Dinh et al. (2016) or Glow by Kingma et al. (2018)
• Masked autoregressive flow by Papamakarios et al. (2017)
• Sylvester normalizing flows by van den Berg et al. (2018)
• Neural autoregressive flow by Huang et al. (2018)
• Continuous flow by Chen et al. (2018)
• Free-form continuous flow by Grathwohl et al. (2018)
• Sum-of-squares polynomial flow by Jaini et al. (2019)
• Residual flow by Chen et al. (2019)
• Neural spline flow by Durkan et al. (2019)
• Unconstrained monotonic neural flow by Wehenkel et al. (2019)
• Circular spline flow by Rezende et al. (2020)
• Augmented flow(s) by Huang et al. (2020)
• Rational-linear spline flow by Dolatabadi et al. (2020)
• Gaussianization flow by Meng et al. (2020)

[bhvieira]

Hello François, thank you for developing, releasing and maintaining zuko, it's an incredible package.
Two missing flows that could be added to the list are

• Sylvester Flows van den Berg et al. (2019)
• LU-Net Chan et al. (2022)

[francois-rozet]

Hi @bhvieira, thank you for the kind words and suggestions! I have added the Sylvester flow to the list. I believe the idea of using LU-decomposition for invertible linear layers comes from Kingma et al. (2018) and later re-used by Durkan et al. (2019). LULinearTransform is already implemented in Zuko actually. However, it could likely be improved by using torch.linalg.lu_factor and torch.linalg.lu_solve.

[bhvieira]

LU-Net does not actually solve the LU decomposition, it's just based on composing two linear layers: one lower triangular and the other upper triangular. The inverse is based on solving a system of linear equations.

[francois-rozet]

Kingma et al. (2018) and Durkan et al. (2019) do exactly what you describe: they parameterize a linear transformation with triangular matrices $L$ and $U$ (and permutation $P$). They can inverse these transformations efficiently because $L$ and $U$ are triangular. The only time the LU-decomposition is performed is at the initialization of the network.