Etymology of "Normalizing Flows"

[chester-tan]

In the "Learn the basics" tutorial, the etymology of the term "normalizing" in "normalizing" flows is explained as: "normalizing refers to the fact that the base distribution is often a (standard) normal distribution.

https://github.com/probabilists/zuko/blob/master/docs/tutorials/basics.ipynb

However in the prototypical(?) paper "Variational Inference with Normalizing Flows" it is explained that "By repeatedly applying the rule for change of variables, the initial density ‘flows’ through the sequence of invertible mappings. At the end of this sequence we obtain a valid probability distribution and hence this type of flow is referred to as a normalizing flow."

https://arxiv.org/abs/1505.05770

Is the etymology of the term "normalizing" in "normalizing" flows then more accurately explained as derived from the ability of the method to produce a "normalized" valid probability distribution instead of being derived from common use of "normal" distributions in the method?

[francois-rozet]

Hello @chester-tan 👋

I think there is indeed a bit of ambiguity here. The term "normalizing" is defined by Tabak et al. (2013) as

Normalizing the data $x_j$ is finding a map $y(x)$ such that the $y_j = y(x_j)$ have a prescribed distribution $\mu(y)$, for which we shall adopt here the isotropic Gaussian [...]

It is not clear to me whether they mean that normalizing means finding a transformation that matches any distribution $\mu(y)$ (with normalized density) or only an isotropic Gaussian (a normal distribution).

However, they later say

There is more than semantics to this rephrasing: normalizing the data is often a goal per se. It allows us, for instance, to compare observations from different datasets, to define robust metrics in phase-space, and to use standard statistical tools, often applicable only to normal distributions.

I therefore prefer the interpretation that normalizing refers to transforming data to a normal distribution. In addition, any distribution has a normalized probability density, even though it might be unknown. In this regard, the data distribution is not less "normalized" than the target distribution. I therefore find this definition confusing.

[chester-tan]

Hi @francois-rozet 👋

Thanks so much for the very helpful reply! 😊

Its etymology does seem a little ambiguous (I guess with many ML terms haha) and it's great to hear your take on it and why it's explained in the (instructive!) tutorials that way.

[francois-rozet]

You are welcome! I will convert this issue to a discussion as it is more appropriate. Feel free to ask more questions 😁