Skip to content

README.md

Latest commit

 

History

History
397 lines (284 loc) · 31.4 KB

README.md

File metadata and controls

397 lines (284 loc) · 31.4 KB

🧠 Brain-Decoding-Guide

License: MIT PRs Welcome

📚 This repo aims to guide researchers who are new to the Brain Decoding field to quickly learn about its techniques, datasets, and applications.

📖 Table of Contents

Introduction · Modalities · Datasets · Surveys · Foundational Works · Recent Advances · Metrics & Tools · Clinical Cases · Learning Resources · Contributing

📌 Introduction

Brain decoding (also referred to as neural decoding) is a computational and neuroscientific technique that extracts meaningful, interpretable information about an individual's subjective mental states, perceptual experiences, cognitive processes, or behavioral intentions directly from recorded brain activity (e.g., fMRI, EEG, MEG, or invasive neural recordings). It relies on machine learning algorithms, statistical modeling, and neuroscientific insights to map patterns of neural activity to specific mental content, with applications in neuroscience research, brain-computer interfaces (BCIs), and clinical neuroscience.

🧬 Brain Signal Modalities

Modality Full Name Spatial Res. Temporal Res. Invasiveness Common Use Cases
fMRI Functional Magnetic Resonance Imaging ~1-3 mm ~1-2 s Non-invasive Visual/Semantic decoding
EEG Electroencephalography ~10 mm ~1 ms Non-invasive Motor imagery, emotion, sleep
MEG Magnetoencephalography ~5 mm ~1 ms Non-invasive Language, auditory processing
ECoG Electrocorticography ~1 cm ~1 ms Invasive Speech BCI, epilepsy
sEEG Stereoelectroencephalography ~5 mm ~1 ms Invasive Deep brain structures
NIRS Near-Infrared Spectroscopy ~10 mm ~100 ms Non-invasive Portable BCI, infants

💡 Tip: fMRI excels at where (spatial), while EEG/MEG excel at when (temporal). Invasive methods (ECoG, sEEG) offer the best of both but require surgery.

Brain Decoding Publications by Signal Modality

📊 Datasets

📂 fMRI Datasets (click to expand)
Task Dataset Signal Description Links
Disease Classification ABIDE-I fMRI 1035 subjects (~111.8 hours); autism detection & gender classification website
Disease Classification ADHD200 fMRI 973 subjects (~129.5 hours); ADHD diagnosis website
Visual Image Decoding NSD (2021) fMRI 7 subjects viewing ~70,000 natural images (~1.5–2TB, application required) website
Visual Image Decoding BOLD5000 (2021) fMRI 4 subjects viewing ~5,000 COCO/ImageNet/SUN images (~150–200GB) website
Visual Image Decoding GOD (2019) fMRI 5 subjects viewing object categories; generic object decoding (~5–10GB) website
Visual Image Decoding THINGS fMRI1 fMRI 3 subjects, 8,640 object images; object recognition & representational geometry website
Visual Image Decoding vim-1 (2011) fMRI 1–2 subjects viewing natural images; visual encoding/decoding (~10GB) website
Video Perception Decoding Algonauts (2021/2023) fMRI 10–30 subjects watching short natural videos; brain encoding challenge (~50–150GB) website
Multi-task / Resting-State Human Connectome Project fMRI 1200 subjects; multi-task & resting-state fMRI (~80–100TB) website
Video Perception Decoding CC2017 (2017) fMRI 3 subjects watching ~3 hours of videos; video brain encoding (~30–40GB) website
Video Perception Decoding BOLD Moments Dataset fMRI 10 subjects watching 1,102 short videos; dynamic visual encoding (~88GB) website
Video Perception Decoding vim-2 (2014) fMRI 3 subjects watching natural videos; visual motion encoding (~50–100GB) website
Facial Expression Decoding NFED (2024) fMRI 5 subjects watching ~1320 facial expression videos (~176GB) website
3D Object Decoding fMRI-Shape fMRI 14 subjects viewing 1600+ 3D objects; 3D shape perception decoding website
3D Object Decoding fMRI-Objaverse fMRI 5 subjects viewing 3,142 3D objects across 117 categories website
Auditory Language Decoding Narratives (2011–2018) fMRI 345 subjects listening to 28 spoken stories; semantic mapping (~132GB) website
Auditory Language Decoding Nature Story Listening (2016) fMRI 11 subjects listening to long-form stories; continuous language encoding website
Language Comprehension MOUS-fMRI fMRI 200+ subjects reading or listening to sentences (well-formed vs scrambled) website
Language Decoding Semantic Listening vs Reading fMRI fMRI during reading and listening to natural stories; modality comparison website
📂 EEG Datasets (click to expand)
Task Dataset Signal Description Links
Alzheimer's / FTD Classification OpenNeuro AD/FTD Dataset EEG 36 AD + 23 FTD patients (~14.9 hours EEG) website
Depression Detection Mumtaz2016 EEG 35 subjects (~20.3 hours EEG) for MDD detection website
Mental Disorder Classification MODMA EEG + Speech Multimodal mental disorder EEG (128+3 channels) and speech website
Seizure Detection SienaScalpEEGDatabase EEG Clinical scalp EEG dataset with seizure annotations website
Seizure Detection CHB-MIT EEG 20 pediatric subjects with epilepsy EEG (~40GB) website
Parkinson's Detection PD31 EEG 31 subjects (~2.5 hours EEG) for Parkinson's detection website
Abnormal EEG Classification TUAB EEG 2000+ subjects, 1000+ hours EEG (application required) website
Visual Image Decoding THINGS-EEG1 (2022) EEG 50 subjects viewing object images; object-level representation (~40GB) website
Visual Image Decoding THINGS-EEG2 (2025) EEG 10 subjects viewing 16,740 images; visual decoding & reconstruction website
Visual Image Decoding Kaneshiro2015 EEG 10 subjects viewing 72 object images; representational geometry website
Visual Image Decoding Grootswagers2019 EEG 16 subjects viewing 200 images; rapid visual categorization website
Visual Image Decoding ImageNet-EEG EEG 16 subjects with 87,850 EEG-image pairs; image reconstruction (~18GB) website
Emotion Recognition MAHNOB-HCI EEG + Video 30 subjects watching emotional videos (partially unavailable) website
Emotion Recognition SEED-DV EEG 15 subjects watching emotional videos; dynamic emotion recognition website
Language / Reading Decoding ZuCo (2018) EEG + Eye-tracking 12 subjects reading text; EEG-to-text decoding (~5–10GB) website
Imagined Speech Decoding Inner Speech Dataset EEG 10 subjects (~9 hours EEG) for imagined speech decoding website
Language / Reading Decoding ChineseEEG-2 EEG 12 subjects performing Chinese reading tasks (~100GB) website
Auditory Language Decoding Broderick2018 EEG 19 subjects listening to natural speech; auditory attention decoding website
Imagined Speech Decoding Chisco EEG >20k high-density EEG samples for imagined speech website
Auditory Language Decoding Brennan-Hale2019 EEG 33 subjects listening to English speech; neural tracking website
Imagined Speech Decoding BCIC2020-3 EEG 15 subjects, 64-channel EEG, 5-class imagined speech website
Imagined Speech Decoding KARA ONE EEG 12 subjects imagined speech dataset (~24GB) website
EEG Event Classification TUEV EEG ~150 hours EEG with event annotations website
Motor Imagery Classification WBCIC_SHU EEG 51 subjects (~34 hours) motor imagery EEG website
Motor Imagery Classification PhysioNet-MI EEG 109 subjects (~10.9 hours) motor imagery EEG website
Motor Imagery Classification BCIC-IV-2a EEG 9 subjects, 22-channel EEG, 4-class motor imagery website
Cognitive Load / Stress MentalArithmetic EEG 36 subjects performing arithmetic tasks for stress classification website
Emotion Recognition SEED (2013) EEG 15 subjects; 3-class emotion recognition (application required) website
Sleep Stage Classification Sleep-EDF (2013) EEG 22 subjects overnight sleep EEG; 5-stage sleep classification (~8.1GB) website
Sleep Stage Classification ISRUC EEG 100 subjects sleep EEG with 5-stage classification website
Multi-task BCI MOABB EEG Benchmark platform integrating 30+ pipelines and 36+ EEG datasets website
📂 MEG Datasets (click to expand)
Task Dataset Signal Description Links
Multi-task / Lifespan Cam-CAN MEG MEG Hundreds of subjects across lifespan; perception, memory, motor (application required) website
Visual Image Decoding THINGS MEG1 MEG 4 subjects, 22,248 image trials; object recognition & representational similarity website
Language Comprehension MOUS-MEG MEG 200+ subjects reading or listening to sentences (well-formed vs scrambled) website
Auditory Language Decoding MEG-MASC MEG 27 subjects listening to speech stimuli; speech neural tracking website
Clinical Neuroimaging OMEGA MEG 444 controls + 200 patients (>150 hours); clinical MEG repository website
Multi-task HCP-MEG MEG MEG subset of HCP with motor, story, working memory tasks website
📂 ECoG / SEEG / MEA Datasets (click to expand)
Task Dataset Signal Description Links
Audiovisual Perception CRCNS ECoG ECoG 21 epilepsy patients performing audiovisual tasks (~8GB) website
Speech Decoding / BCI Metzger2023 ECoG Single-subject speech neuroprosthesis ECoG dataset website
Language / Reading Decoding Verwoert2022 sEEG 54–127 epilepsy patients performing reading tasks website
Speech Decoding / BCI Willett2023 MEA (intracortical) Neural recordings for speech prosthesis with 12,100 spoken sentences website
📂 Multimodal Datasets (click to expand)
Task Dataset Signal Description Links
Motor Imagery / Decoding SomatoMotor EEG + MEG 5 subjects (~0.7 hours) EEG+MEG motor task dataset website
Video / Multimodal Perception CineBrain EEG + fMRI 6 subjects (~6 hours) multimodal dataset with video stimuli website
Resting-State / Connectivity LEMON EEG + fMRI 220 subjects (~39.6 hours) resting-state EEG+fMRI dataset website
Natural Viewing / Visual Encoding Nat-View EEG + fMRI 22 subjects (~42.8 hours) natural viewing EEG+fMRI dataset website
Language Comprehension SMN4Lang MEG + fMRI 12 subjects (~70.4 hours) multimodal language dataset website
Multi-task BCI L-mind EEG + fNIRS + PPG 12 subjects multimodal dataset with 23,928 instruction-based samples website
Sleep Stage Classification CAP EEG + EOG + EMG Sleep dataset with CAP annotations including normal and pathological recordings website
Sleep Stage Classification MASS EEG + EOG + EMG Large-scale multimodal sleep dataset with 200+ subjects website

📑 Key Surveys

Year Title Venue Highlights
2025 A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli TPAMI Dataset/ROI summaries, model taxonomy (end-to-end, pre-trained, LLM-centric)
2025 Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding TMLR Encoding + decoding, DL-brain alignment [Code]
2025 Transformer-based EEG Decoding: A Survey ArXiv 200+ papers on Transformer for EEG (2019-2024)
2025 Brain Foundation Models: A Survey ArXiv Foundation models for neural signals, pre-training paradigms
2024 Deep Representation Learning for EEG-based BCIs: A Review ArXiv Autoencoders, SSL, foundation models for EEG
2022 fMRI Brain Decoding and Its Applications in BCI: A Survey Brain Classic ML to deep learning evolution

📜 Foundational Works (Pre-2023)

Milestone papers that established the field.

Year Title Task Feature Links
2016 Natural Speech Reveals the Semantic Maps that Tile Human Cerebral Cortex Semantic Cortical semantic atlas
2017 Deep Learning with Convolutional Neural Networks for EEG Decoding and Visualization Motor Interpretable filters [Code]
2018 EEGNet: A Compact Convolutional Neural Network for EEG-based BCIs Motor BCI baseline [Code]
2019 Deep Image Reconstruction from Human Brain Activity Visual Feature optimization

⚙️ Recent Advances & Core Algorithms

High-impact papers from 2023-2025.

Modern brain decoding systems are built on three complementary AI stacks: an Encoder Stack that learns neural representations via self-supervised pretraining (MAE, contrastive learning) and aligns them to shared cross-modal embedding spaces; a Decoder Stack that reconstructs stimuli using conditioned diffusion models or autoregressive transformers with brain-signal serialization; and a "Unified" Stack of brain foundation models that integrate spatio-temporal transformers, cross-modal knowledge distillation, and parameter-efficient fine-tuning (LoRA/PEFT) for generalizable, multi-subject decoding.

Core AI Technology Stack for Brain Decoding Figure: Core AI Technology Stack System for Brain Decoding — covering the Encoder Stack (neural representation & cross-modal alignment), Decoder Stack (generative decoding & reconstruction), and the Unified Stack (brain foundation models & multimodal fusion).

🖼️ Visual Reconstruction

📂 fMRI → Image (click to expand)
Year Title Arch Feature Links
2023 High-Resolution Image Reconstruction with Latent Diffusion Models from Human Brain Activity Diffusion Direct fMRI-to-LDM mapping without fine-tuning [Code]
2023 Seeing Beyond the Brain: MinD-Vis Diffusion Large-scale resting-state fMRI pre-training + sparse coding [Code]
2023 Brain-Diffuser: Natural Scene Reconstruction from fMRI Signals Diffusion VDVAE low-level + CLIP/BLIP high-level dual-pipeline reconstruction [Code]
2023 Reconstructing the Mind's Eye: MindEye Diffusion Dual-path: contrastive retrieval + diffusion prior [Code]
2023 UniBrain: Unify Image Reconstruction and Captioning from Human Brain Activity Diffusion First unified fMRI-to-image + captioning in a single LDM model
2024 MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data Diffusion Cross-subject transfer via functional alignment; only 1hr data needed [Code] [Website]
2024 MindBridge: A Cross-Subject Brain Decoding Framework Diffusion Single model for multi-subject; bio-inspired aggregation [Code]
📂 EEG → Image (click to expand)
Year Title Arch Feature Links
2024 DreamDiffusion: Generating High-Quality Images from Brain EEG Signals Diffusion Temporal masking pre-train + CLIP alignment; first EEG-to-image [Code]
2024 Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (ATM) Diffusion Adaptive Thinking Mapper (ATM) encoder; zero-shot cross-subject reconstruction [Code]
📂 EEG → Video (click to expand)
Year Title Arch Feature Links
2024 EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals Diffusion Seq2Seq EEG encoder; first EEG-to-video reconstruction; 79.8% semantic accuracy [Code]
📂 fMRI → Video (click to expand)
Year Title Arch Feature Links
2023 Cinematic Mindscapes: High-Quality Video Reconstruction from Brain Activity Diffusion Spatiotemporal attention + contrastive learning; arbitrary frame-rate [Code] [Website]
2025 Animate Your Thoughts: Reconstruction of Dynamic Natural Vision from Human Brain Activity Diffusion Decouple fMRI signals into semantic, structural, and motion features, then decode them to each frame of synthesized GIFs [Code]

🗣️ Speech & Language Decoding

📂 Invasive Speech — ECoG / Intracortical (click to expand)
Year Title Arch Feature Links
2023 A High-Performance Speech Neuroprosthesis RNN 62 wpm; first large-vocab decoding (125k words) [Code]
2023 A High-Performance Neuroprosthesis for Speech Decoding and Avatar Control RNN Real-time avatar control with facial expression + speech
2025 A Streaming Brain-to-Voice Neuroprosthesis RNN-Transducer 80ms streaming decoding; real-time speech synthesis [Code]
📂 Non-invasive Semantic — fMRI / EEG / MEG (click to expand)
Year Title Arch Feature Links
2023 Semantic Reconstruction of Continuous Language from Non-invasive Brain Recordings Transformer GPT autoregressive decoding + beam search; multi-cortex support
2024 DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation Transformer Discrete codebook alignment to LLM; no word-level gaze annotation
2025 Decoding Individual Words from Non-invasive Brain Recordings (EEG/MEG) Transformer Deep learning pipeline decoding individual words from EEG and MEG signals

🎯 Motor & Intention Decoding

📂 Motor Imagery Papers (click to expand)
Year Title Arch Feature Links
2024 CTNet: A Convolutional Transformer Network for EEG-based Motor Imagery Classification CNN-Transformer CNN local features + Transformer global dependencies
2023 EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization Conformer Conv + self-attention; CAM-based topographic visualization [Code]
2022 ATCNet: Attention Temporal Convolutional Network for EEG-based Motor Imagery Classification TCN Sliding window + multi-head attention + TCN residual [Code]

🧩 Brain Foundation Models

📂 Foundation Model Papers (click to expand)
Year Title Arch Feature Links
2024 LaBraM: Large Brain Model for Learning Generic Representations with Tremendous EEG Data Transformer Pre-trained on ~2,500 hours EEG across 20+ datasets; vector-quantized neural spectrum prediction [Code]
2023 BRANT: Foundation Model for Intracortical Neural Signal Transformer Spatiotemporal Transformer pre-trained on large-scale intracortical data

📏 Metrics & Tools

Evaluation Metrics

Category Metric Description
Encoding Pearson r, R² Correlation between predicted and actual brain activity
Low-level Reconstruction PixCorr, SSIM, PSNR Pixel-level similarity
High-level Reconstruction CLIP Score, Inception Score Semantic/perceptual similarity
Classification Accuracy, F1, AUC Standard classification metrics
Retrieval Top-k Accuracy, MRR Retrieval-based evaluation

Software & Libraries

Tool Description Link
MNE-Python MEG, EEG, sEEG, ECoG, NIRS analysis [Website] [GitHub]
Nilearn Statistical learning on fMRI data [Website] [GitHub]
Braindecode Deep learning for EEG/ECoG/MEG decoding; EEGNet, ShallowNet, etc. [Website] [GitHub]
TorchEEG PyTorch library for EEG processing & models [GitHub]
Net2Brain Compare DNN activations with brain activity (RSA, encoding) [GitHub]
Neural_Decoding Classic + DL decoders (Kalman, Wiener, LSTM, etc.) [GitHub]
PyCortex fMRI visualization on cortical surface [GitHub]
RSA Toolbox Representational Similarity Analysis [GitHub]

Benchmark Platforms

Platform Description Link
Algonauts Project Annual challenge for predicting brain responses to visual stimuli [Website]
Brain-Score Benchmark for comparing DNNs with primate visual cortex [Website] [GitHub]
MOABB Mother of All BCI Benchmarks; 36 EEG datasets, 30 pipelines [Website] [GitHub]

🏥 Clinical Application Cases

Recent breakthroughs demonstrating real-world clinical impact.

Case Year Description Links
Synchron & Apple: Thought-Controlled iPad 2025 ALS patient controlled iPad via Stentrode implant + Apple BCI HID protocol—navigating apps, composing texts using only thoughts [News] [Video]
Neuralink Telepathy 2024 First Neuralink human implant; quadriplegic patient played chess & Civilization VI via cursor control using thoughts alone [News] [Video]
UC Davis ALS Speech BCI 2024 Restored speech for ALS patient with >97% accuracy; preserved voice identity using high-density ECoG [Press] [Paper]

📚 Learning Resources

📺 Video Tutorials & Courses

📂 Video Tutorials (click to expand)
Resource Description Link
Neuromatch Academy World-class open course on computational neuroscience; encoding/decoding basics [Website] [YouTube] [Bilibili]
INCF: Deep Learning in Neuroscience Beginner-level DL for neuroscience applications [Website]

📖 Textbooks & Reading

📂 Textbooks & Reading (click to expand)
Resource Description Link
Deep Learning (Goodfellow et al.) Deep learning bible; free online [Website]
Awesome-Brain-Encoding-Decoding Curated paper list [GitHub]

🌐 Communities

📂 Communities (click to expand)
Community Description Link
NeuroAI WeChat Group Chinese community for brain + AI research Contact via WeChat number MobiusAI
BCI Society International BCI research community [Website]
OHBM Organization for Human Brain Mapping [Website]

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. See CONTRIBUTING.md for guidelines.

If you find this guide helpful, please consider giving it a ⭐!