A question about I-JEPA on SAR data?

[waterdisappear]

Hello! I thoroughly enjoyed reading your paper, "AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities," and I am truly impressed by the depth of your research and its contributions to the field.

I have a question about I-JEPA on SAR data:
When I try to implement JEPA (https://github.com/facebookresearch/ijepa) from on SAR data, i.e., using a learnable Transformer with EMA to encode the target features, I experience feature crashes; the loss down drops to very small, close to 1e-5, within 3 epochs, but the performance of the downstream task are very bad. I've been confused about whether it's due to the amount of data or the SAR image noise effect; I'm curious if you've encountered a similar problem and can provide some valuable advice.

Thanks for your time and for advancing research in this area. I look forward to your response.

Best regards,
Weijie Li

[gastruc]

Hi,
Collapsing mode are indeed the main drawback of JEPA pretraining. From my experience, AnySat is really stable and I've only experienced a few times collapsing modes during my different experiences. I am not really able to say why it is that stable but my intuitions would be:

• the contrastive loss that encourage diversity in mid-term features
• the amount of data
• the masking ratio parameters
  I have never launched a pretraining with only SAR data so I don't have much insights.