Foundation Model Adapters for Spectrogram Latent Features Extraction

[alexandraChangLi]

Title

Foundation Model Adapters for Spectrogram Latent Features Extraction

Leaders

Chang Li @alexandraChangLi

Collaborators

1. Yamin Li @soupeeli

Project Description

Topic: Leveraging foundation models’ pre-trained knowledge to conduct spectrogram analysis of EEG and fMRI resting state data of healthy subjects.

Objective: Differentiate drowsy and alertness states based on EEG and fMRI spectrogram features.

Potential Contribution: Adapt foundation models’ robust, pre-trained knowledge of the brain-imaging domain can potentially generate high-quality latent space with good discriminability among subjects and brain states.

Get Started: Download the sample code and data. Follow the tutorial in this readme file to set up environments and run the basic pipeline. Integrate foundation models for feature extraction. Visualization and analysis.

Key Resources: Background theory: foundation models’ official tutorials, vision detection models. Implementation: PyTorch documentations. ChatGPT is your good friend if you use it wisely!

Link to project repository/sources

https://github.com/neurdylab/FoundationModelAdapters

Concerete Goals with Specific Tasks for Brainhack Vanderbilt 2025

1. Get a basic understanding of brain imaging spectrograms (not computationally intensive).
   a. Understand how brain imaging spectrograms are generated.
   b. Understand the differences between EEG and fMRI spectrograms, as well as the pros and cons of these two modalities.
   c. Differentiate drowsy and alert states from EEG and fMRI spectrograms manually, and think about what might contribute to the differences in latent space.

2. Compare different foundation models’ abilities in capturing brain images’ intrinsic information and the quality of latent space (GPU preferred; CPU with a large memory is also fine!).
   a. Experiment with the CLIP feature extractor in the sample code.
   b. Integrate visual foundation models with open-source checkpoints/APIs (e.g. DALL-E, SAM) as feature extractors.
   c. Visualize the features with UMAP (sample code provided) or other methods.

3. Implement fine-tuning blocks for downstream tasks (e.g. classification) based on the latent space.
   a. Experiment with the adapter for CLIP features in the sample code. Experiment with the MLP predictor for alertness/drowsiness classification.
   b. Modify the adapter layers to enhance discriminability between features from different brain states. Modify the MLP predictor to enhance model performance.
   c. Experiment with features from other foundation models.

Good first issues

1. Issue one: Set up environments and run the model pipeline.

2. Issue two: How to extract visual features from spectrogram images.

3. Issue three: How to enhance discriminability between different brain states in the latent feature space.

Skills

1. Able to set up Python deep learning environment (IDE, Anaconda). Have Python programming experience or at least intermediate knowledge level of other languages.
2. A beginner-level understanding of visual/language foundation models, e.g. CLIP, DALL-E, SAM, no prior experience required.
3. Intermediate deep learning experience preferred.
4. Beginner-level Git knowledge required. Intermediate Git experience preferred.

Onboarding documentation

GitHub: https://github.com/neurdylab/FoundationModelAdapters
Readme: https://github.com/neurdylab/FoundationModelAdapters/blob/main/README.md

What will participants learn?

1. Integration of foundation models into deep learning pipeline.
2. Basic EEG/fMRI spectrogram interpretation.
3. Understanding of EEG/fMRI latent space.

Public data to use

Paired EEG and fMRI spectrograms are generated from part of the internal dataset of Neurdylab.

Number of collaborators

3

Credit to collaborators

Collaborators contributing to method design/experiments/visualizations/writings will become coauthors in follow-up publications.

Image

Project Summary

Leveraging foundation models’ pre-trained knowledge to conduct spectrogram analysis of EEG and fMRI resting state data of healthy subjects.

Type

method_development

Development status

1_basic structure

Topic

deep_learning

Tools

MNE, Pytorch, other

Programming language

Python

Modalities

EEG, fMRI

Git skills

1_commit_push

Anything else?

Metadata

1. Computational Resources: The codebase is implemented in both the CPU and GPU versions. Change the arguments in the running scripts for switching computational devices. Sample data size: < 61M. Memory usage in training: 3000MiB.
2. Code Implementation: The pipeline is implemented following [1][2][3].
3. Data Preparation: EEG spectrograms are generated following the work of [4], and fMRI spectrograms are generated following the work of [5]. Brain states ground truth is computed via the VIGALL algorithm [6]. Ground truth label in dataloader: eeg_index.

References
[1] Misra I, Girdhar R, Joulin A. An end-to-end transformer model for 3d object detection[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 2906-2917.
[2] Lu Y, Xu C, Wei X, et al. Open-vocabulary point-cloud object detection without 3d annotation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 1190-1199.
[3] Gao P, Geng S, Zhang R, et al. Clip-adapter: Better vision-language models with feature adapters[J]. International Journal of Computer Vision, 2024, 132(2): 581-595.
[4] Prerau M J, Brown R E, Bianchi M T, et al. Sleep neurophysiological dynamics through the lens of multitaper spectral analysis[J]. Physiology, 2017, 32(1): 60-92.
[5] Song C, Boly M, Tagliazucchi E, et al. fMRI spectral signatures of sleep[J]. Proceedings of the National Academy of Sciences, 2022, 119(30): e2016732119.
[6] Olbrich S, Fischer M M, Sander C, et al. Objective markers for sleep propensity: comparison between the Multiple Sleep Latency Test and the Vigilance Algorithm Leipzig[J]. Journal of sleep research, 2015, 24(4): 450-457.

Things to do after the project is submitted and ready to review.

• Add a comment below the main post of your issue saying: Hi @brainhack-vandy/project-monitors my project is ready!

[alexandraChangLi]

Hi! My project is ready👀!