What is their "Hilbert space" and how strong are the assumptions?

[el-hult]

Following up on my confusion on the seminar today. I just can't seem to understand how the paper is supposed to work out with their generalities.

Therefore, I write this post in hope that someone can chime in or tell me where I am confused.

So we have random elements x in some Hilbert space. For every vector, we can construct the tensor product xx^T which by definition is a linear operator that takes a vector y \mapsto x<x,y> where <.,.> is the inner product of the Hilbert space.

This operator is self-adjoint, since <y,xx^Tz> = <xx^Ty,z>. It has the operator norm bounded via Schwartz inequality |xx^Tv|/|v| <= |x|^2.

Then we compute its expectation, via some suitable integral E[xx^T].
I think (hope) that the result will be self-adjoint and bounded still, though those details are beyond me. Since it is linear and bounded, it is compact. Since it is self-adjoint, it has a countable specturm with countable orthonormizable eigenvectors. The details can be found in many places. E.g. Theorem 2.39 in ¹. So I interpret the notation in the paper as if V^T is a linear operator from the Hilbert space to the coordinate representation in l^2, the hilbert space of square-summable real number sequences. So the vectors z in the articles are random elements in l^2.

Don't we have very strong assumptions about them now? Since every realization of z must be square summable, and its components must at the same time be independent.

I can come up with the following case: let the ith component of z have support on [-1/n,1/n] then, at worst, the realizations of z will have norm \sum 1/n^2, which is finite. But this says a lot about how well-excited the different eigenspaces of E[xx^T] are excited.

Furthermore, they do use the notation \lambda ^T z, so that means that \lambda^T and z are in dual spaces. And since z must be in l^2, \lambda must be too. And that is, according to the text, the Hilbert space H.

Notably, the wikipedia text on separable hilbert spaces notes that it was common before to always refer to l^2 as the hilbert space. So I guess this is what they mean in the article...

tl;dr

1. Do you agree that the authors implicitly mean l^2 when they say "Hilbert space?"
2. Do you also think that we have very strong restrictions on x, coming from the fact that the components of z are subgaussian independent, but the element is in l^2?

Footnotes

1. M. Carrasco, J.-P. Florens, and E. Renault, “Chapter 77 Linear Inverse
   Problems in Structural Econometrics Estimation Based on Spectral Decom-
   position and Regularization,” in Handbook of Econometrics, vol. 6, pp. 5633–
   5751, Elsevier, 2007. ↩

[mcpca]

Hey!

The covariance is probably best defined as is done in the wikipedia page https://en.wikipedia.org/wiki/Covariance_operator so that one just deals with Lebesgue integrals, this was the first point of confusion for me. Then adjointedness is direct (and boundedness seems to be tacitly assumed).

Regarding the operator V, I cannot see any other interpretation than the one you give (V : \ell^2 \to H) and then z \in \ell^2 necessarily.
So either the \lambda \in H is a typo, or the authors mean they map z back to H in some way to be able to write \lambda^T z, or they don't really differentiate between H and \ell^2.
I am more inclined towards the first hypothesis since (1) then it is somewhat clear what they mean by components of z and (2) there can conceivably many ways to isometrically map H to \ell^2. Maybe all these isometries preserve subgaussianity but \sigma_x would change.

I do agree that it seems to impose strong requirements on the components of z (and so on x).
Overall the literature on this seems to be sparse, I am a bit sad that the authors do not seem to include a reference?

I have a more basic question related to this that we also did not have time to get to yesterday: When is the infinite dimensional case actually relevant in practice? I am vaguely familiar with kernel methods where I know infinite dimensional Hilbert spaces can show up but is that related to the setup of the authors (infinite dimensional covariates)?

[el-hult]

Good points. Thank you. :)

Sure, there is a link to the kernel litterature, I suppose. On the other hand, the whole point with the kernel methods is that you never work in the infinite dimensional setting, but rather only with a finite number of inner products on that space. So it is not clear to me how they plan to make that transition.

Rather, I suppose that they had time series data in mind. If the hilbert space indeed is \ell^2, you could consider singals with finite energy, and work with those for prediction or classification and so on. That would make sense to me at least.

[antonior92]

When is the infinite dimensional case actually relevant in practice?

Besides the link with more traditional kernel methods. .. One case of interest that I can think of: Neural networks as we make them infinitely wide tend to a linear regression in an infinitely-dimensional kernel space (i.e., as in the NTK paper). So allowing for the infinite-dimensional case could make it easier to analyze this case as well...