Mapping from paper to code

[sfalkena]

Hi! I am trying to map the explanation of the projectors in the paper to their implementations in the repository, but I am stuck on mapping one to the other. In specific, I am trying to find the described steps from section 3.1 in the repository. From the paper I understand that we have 3 different granularities: tile (with dimensions S/R), patch (not sure about dimension) and sub-patch (dimension delta_m). I understand from the paper that patch size p is kept constant in meters, so that every modality covers the same geospatial area. However, I can't find in the code where you do the mapping from tile to patch (assuming that "tile" is our input image). Since you keep the size in meters constant, somewhere a mapping needs to be done that incorporates the resolution, but I fail to find that.

Some questions:

1. II was assuming that delta_m from the paper is mapped to patch_size in the repo, is that correct?
2. If so, why do some modalities in AnySat.yaml have patch_size defined and others don't? How are you handling the different resolutions for Sentinel 2 channels for example?
3. What is the scale parameter in the AnyModule class? I dont seem to see that in the paper. Does this define our "zoom"?
4. In the dataset configs, I see the parameter num_patches. Is this the number of subpatches in a tile? How is this number calculated?

In general, could you help me get up to speed in understanding the architecture by pointing me to specific important parts of the code?

Thanks!

[sfalkena]

I think I understand more of it now, so let me answer part of my own questions and ask some clarifying questions:

1. This seems to be correct indeed, although readme mentions "Patch size (in m)", while I believe delta_m is in pixels.
2. You are using curated datasets, which have already resampled all bands to the same 10m resolution. Resolution and patch_size are only defined in AnySat.yaml for non-temporal modalities, since they use PatchMLPMulti as a projector. Temporal modalities use PatchLTAEMulti.
3. I read in the section "Position Encodings" in the appendix that "P is randomly chosen for each batch during training". Is P calculated as:

   

        $$P = scale \cdot R_m \cdot \delta_m$$    

4. I guess my assumption was right and

   

        $$num\_patches =\left( \frac{S}{R_m \cdot \delta_m}\right)^2 = \left( \frac{tile\_size\_meters}{resolution \cdot patch\_size\_pixels}\right)^2 $$    

5. Based on what metric is the scale chosen during finetuning? Again taking Pastis as example, the scale is taken as 4 in the finetuning step.
6. How is the size of the subpatch in meters chosen? Right now it is 10 meters, but I would be curious to know if this was chosen based on some experiment.

[gastruc]

Hi,
Thank you for your deep interest in our work!

1. patch_size argument of anysat is indeed in meters and the model will retrieve ∆m with the formula ∆m = P/Rm.
2. Your are right for the second point, this is a bad name for the variable. It represents the size in pixels of supatches.
3. During pretraining, we sample a P among listed possibilities to train our network in different settings. Your formula seems correct
4. Yes
5. The patch size is chosen to fit any observation in the GPU memory as it is allows to "break" the quadratism of self attention in the number of tokens by spliiting it. My method was taking the lower one that fits into my GPU. Also, is has to be a multiple of the resolutions of your modalities.
6. Subpatches differs in size according to modalities. It is 10m for the major part of the modalities but can be lower (2m for high resolution images), or bigger (30m for landsat 7 for example). These numbers reflects either the size of 1 pixel if we consider that the data does not need to be spatially reduced, or 10x10 pixels if we consider that it should be spatially reduced. We have not investigated different methods.