🧠 Brain-Decoding-Guide

📚 This repo aims to guide researchers who are new to the Brain Decoding field to quickly learn about its techniques, datasets, and applications.

---

📖 Table of Contents

Introduction · Modalities · Datasets · Surveys · Foundational Works · Recent Advances · Metrics & Tools · Clinical Cases · Learning Resources · Contributing

---

📌 Introduction

Brain decoding (also referred to as neural decoding) is a computational and neuroscientific technique that extracts meaningful, interpretable information about an individual's subjective mental states, perceptual experiences, cognitive processes, or behavioral intentions directly from recorded brain activity (e.g., fMRI, EEG, MEG, or invasive neural recordings). It relies on machine learning algorithms, statistical modeling, and neuroscientific insights to map patterns of neural activity to specific mental content, with applications in neuroscience research, brain-computer interfaces (BCIs), and clinical neuroscience.

---

🧬 Brain Signal Modalities

Modality	Full Name	Spatial Res.	Temporal Res.	Invasiveness	Common Use Cases
fMRI	Functional Magnetic Resonance Imaging	~1-3 mm	~1-2 s	Non-invasive	Visual/Semantic decoding
EEG	Electroencephalography	~10 mm	~1 ms	Non-invasive	Motor imagery, emotion, sleep
MEG	Magnetoencephalography	~5 mm	~1 ms	Non-invasive	Language, auditory processing
ECoG	Electrocorticography	~1 cm	~1 ms	Invasive	Speech BCI, epilepsy
sEEG	Stereoelectroencephalography	~5 mm	~1 ms	Invasive	Deep brain structures
NIRS	Near-Infrared Spectroscopy	~10 mm	~100 ms	Non-invasive	Portable BCI, infants

💡 Tip: fMRI excels at where (spatial), while EEG/MEG excel at when (temporal). Invasive methods (ECoG, sEEG) offer the best of both but require surgery.

---

📊 Datasets

📂 fMRI Datasets (click to expand)

Task	Dataset	Signal	Description	Links
Disease Classification	ABIDE-I	fMRI	1035 subjects (~111.8 hours); autism detection & gender classification	website
Disease Classification	ADHD200	fMRI	973 subjects (~129.5 hours); ADHD diagnosis	website
Visual Image Decoding	NSD (2021)	fMRI	7 subjects viewing ~70,000 natural images (~1.5–2TB, application required)	website
Visual Image Decoding	BOLD5000 (2021)	fMRI	4 subjects viewing ~5,000 COCO/ImageNet/SUN images (~150–200GB)	website
Visual Image Decoding	GOD (2019)	fMRI	5 subjects viewing object categories; generic object decoding (~5–10GB)	website
Visual Image Decoding	THINGS fMRI1	fMRI	3 subjects, 8,640 object images; object recognition & representational geometry	website
Visual Image Decoding	vim-1 (2011)	fMRI	1–2 subjects viewing natural images; visual encoding/decoding (~10GB)	website
Video Perception Decoding	Algonauts (2021/2023)	fMRI	10–30 subjects watching short natural videos; brain encoding challenge (~50–150GB)	website
Multi-task / Resting-State	Human Connectome Project	fMRI	1200 subjects; multi-task & resting-state fMRI (~80–100TB)	website
Video Perception Decoding	CC2017 (2017)	fMRI	3 subjects watching ~3 hours of videos; video brain encoding (~30–40GB)	website
Video Perception Decoding	BOLD Moments Dataset	fMRI	10 subjects watching 1,102 short videos; dynamic visual encoding (~88GB)	website
Video Perception Decoding	vim-2 (2014)	fMRI	3 subjects watching natural videos; visual motion encoding (~50–100GB)	website
Facial Expression Decoding	NFED (2024)	fMRI	5 subjects watching ~1320 facial expression videos (~176GB)	website
3D Object Decoding	fMRI-Shape	fMRI	14 subjects viewing 1600+ 3D objects; 3D shape perception decoding	website
3D Object Decoding	fMRI-Objaverse	fMRI	5 subjects viewing 3,142 3D objects across 117 categories	website
Auditory Language Decoding	Narratives (2011–2018)	fMRI	345 subjects listening to 28 spoken stories; semantic mapping (~132GB)	website
Auditory Language Decoding	Nature Story Listening (2016)	fMRI	11 subjects listening to long-form stories; continuous language encoding	website
Language Comprehension	MOUS-fMRI	fMRI	200+ subjects reading or listening to sentences (well-formed vs scrambled)	website
Language Decoding	Semantic Listening vs Reading	fMRI	fMRI during reading and listening to natural stories; modality comparison	website

📂 EEG Datasets (click to expand)

Task	Dataset	Signal	Description	Links
Alzheimer's / FTD Classification	OpenNeuro AD/FTD Dataset	EEG	36 AD + 23 FTD patients (~14.9 hours EEG)	website
Depression Detection	Mumtaz2016	EEG	35 subjects (~20.3 hours EEG) for MDD detection	website
Mental Disorder Classification	MODMA	EEG + Speech	Multimodal mental disorder EEG (128+3 channels) and speech	website
Seizure Detection	SienaScalpEEGDatabase	EEG	Clinical scalp EEG dataset with seizure annotations	website
Seizure Detection	CHB-MIT	EEG	20 pediatric subjects with epilepsy EEG (~40GB)	website
Parkinson's Detection	PD31	EEG	31 subjects (~2.5 hours EEG) for Parkinson's detection	website
Abnormal EEG Classification	TUAB	EEG	2000+ subjects, 1000+ hours EEG (application required)	website
Visual Image Decoding	THINGS-EEG1 (2022)	EEG	50 subjects viewing object images; object-level representation (~40GB)	website
Visual Image Decoding	THINGS-EEG2 (2025)	EEG	10 subjects viewing 16,740 images; visual decoding & reconstruction	website
Visual Image Decoding	Kaneshiro2015	EEG	10 subjects viewing 72 object images; representational geometry	website
Visual Image Decoding	Grootswagers2019	EEG	16 subjects viewing 200 images; rapid visual categorization	website
Visual Image Decoding	ImageNet-EEG	EEG	16 subjects with 87,850 EEG-image pairs; image reconstruction (~18GB)	website
Emotion Recognition	MAHNOB-HCI	EEG + Video	30 subjects watching emotional videos (partially unavailable)	website
Emotion Recognition	SEED-DV	EEG	15 subjects watching emotional videos; dynamic emotion recognition	website
Language / Reading Decoding	ZuCo (2018)	EEG + Eye-tracking	12 subjects reading text; EEG-to-text decoding (~5–10GB)	website
Imagined Speech Decoding	Inner Speech Dataset	EEG	10 subjects (~9 hours EEG) for imagined speech decoding	website
Language / Reading Decoding	ChineseEEG-2	EEG	12 subjects performing Chinese reading tasks (~100GB)	website
Auditory Language Decoding	Broderick2018	EEG	19 subjects listening to natural speech; auditory attention decoding	website
Imagined Speech Decoding	Chisco	EEG	>20k high-density EEG samples for imagined speech	website
Auditory Language Decoding	Brennan-Hale2019	EEG	33 subjects listening to English speech; neural tracking	website
Imagined Speech Decoding	BCIC2020-3	EEG	15 subjects, 64-channel EEG, 5-class imagined speech	website
Imagined Speech Decoding	KARA ONE	EEG	12 subjects imagined speech dataset (~24GB)	website
EEG Event Classification	TUEV	EEG	~150 hours EEG with event annotations	website
Motor Imagery Classification	WBCIC_SHU	EEG	51 subjects (~34 hours) motor imagery EEG	website
Motor Imagery Classification	PhysioNet-MI	EEG	109 subjects (~10.9 hours) motor imagery EEG	website
Motor Imagery Classification	BCIC-IV-2a	EEG	9 subjects, 22-channel EEG, 4-class motor imagery	website
Cognitive Load / Stress	MentalArithmetic	EEG	36 subjects performing arithmetic tasks for stress classification	website
Emotion Recognition	SEED (2013)	EEG	15 subjects; 3-class emotion recognition (application required)	website
Sleep Stage Classification	Sleep-EDF (2013)	EEG	22 subjects overnight sleep EEG; 5-stage sleep classification (~8.1GB)	website
Sleep Stage Classification	ISRUC	EEG	100 subjects sleep EEG with 5-stage classification	website
Multi-task BCI	MOABB	EEG	Benchmark platform integrating 30+ pipelines and 36+ EEG datasets	website

📂 MEG Datasets (click to expand)

Task	Dataset	Signal	Description	Links
Multi-task / Lifespan	Cam-CAN MEG	MEG	Hundreds of subjects across lifespan; perception, memory, motor (application required)	website
Visual Image Decoding	THINGS MEG1	MEG	4 subjects, 22,248 image trials; object recognition & representational similarity	website
Language Comprehension	MOUS-MEG	MEG	200+ subjects reading or listening to sentences (well-formed vs scrambled)	website
Auditory Language Decoding	MEG-MASC	MEG	27 subjects listening to speech stimuli; speech neural tracking	website
Clinical Neuroimaging	OMEGA	MEG	444 controls + 200 patients (>150 hours); clinical MEG repository	website
Multi-task	HCP-MEG	MEG	MEG subset of HCP with motor, story, working memory tasks	website

📂 ECoG / SEEG / MEA Datasets (click to expand)

Task	Dataset	Signal	Description	Links
Audiovisual Perception	CRCNS ECoG	ECoG	21 epilepsy patients performing audiovisual tasks (~8GB)	website
Speech Decoding / BCI	Metzger2023	ECoG	Single-subject speech neuroprosthesis ECoG dataset	website
Language / Reading Decoding	Verwoert2022	sEEG	54–127 epilepsy patients performing reading tasks	website
Speech Decoding / BCI	Willett2023	MEA (intracortical)	Neural recordings for speech prosthesis with 12,100 spoken sentences	website

📂 Multimodal Datasets (click to expand)

Task	Dataset	Signal	Description	Links
Motor Imagery / Decoding	SomatoMotor	EEG + MEG	5 subjects (~0.7 hours) EEG+MEG motor task dataset	website
Video / Multimodal Perception	CineBrain	EEG + fMRI	6 subjects (~6 hours) multimodal dataset with video stimuli	website
Resting-State / Connectivity	LEMON	EEG + fMRI	220 subjects (~39.6 hours) resting-state EEG+fMRI dataset	website
Natural Viewing / Visual Encoding	Nat-View	EEG + fMRI	22 subjects (~42.8 hours) natural viewing EEG+fMRI dataset	website
Language Comprehension	SMN4Lang	MEG + fMRI	12 subjects (~70.4 hours) multimodal language dataset	website
Multi-task BCI	L-mind	EEG + fNIRS + PPG	12 subjects multimodal dataset with 23,928 instruction-based samples	website
Sleep Stage Classification	CAP	EEG + EOG + EMG	Sleep dataset with CAP annotations including normal and pathological recordings	website
Sleep Stage Classification	MASS	EEG + EOG + EMG	Large-scale multimodal sleep dataset with 200+ subjects	website

---

📑 Key Surveys

Year	Title	Venue	Highlights
2025	A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli	TPAMI	Dataset/ROI summaries, model taxonomy (end-to-end, pre-trained, LLM-centric)
2025	Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding	TMLR	Encoding + decoding, DL-brain alignment [Code]
2025	Transformer-based EEG Decoding: A Survey	ArXiv	200+ papers on Transformer for EEG (2019-2024)
2025	Brain Foundation Models: A Survey	ArXiv	Foundation models for neural signals, pre-training paradigms
2024	Deep Representation Learning for EEG-based BCIs: A Review	ArXiv	Autoencoders, SSL, foundation models for EEG
2022	fMRI Brain Decoding and Its Applications in BCI: A Survey	Brain	Classic ML to deep learning evolution

---

📜 Foundational Works (Pre-2023)

Milestone papers that established the field.

Year	Title	Task	Feature	Links
2016	Natural Speech Reveals the Semantic Maps that Tile Human Cerebral Cortex	Semantic	Cortical semantic atlas
2017	Deep Learning with Convolutional Neural Networks for EEG Decoding and Visualization	Motor	Interpretable filters	[Code]
2018	EEGNet: A Compact Convolutional Neural Network for EEG-based BCIs	Motor	BCI baseline	[Code]
2019	Deep Image Reconstruction from Human Brain Activity	Visual	Feature optimization

---

⚙️ Recent Advances & Core Algorithms

High-impact papers from 2023-2025.

Modern brain decoding systems are built on three complementary AI stacks: an Encoder Stack that learns neural representations via self-supervised pretraining (MAE, contrastive learning) and aligns them to shared cross-modal embedding spaces; a Decoder Stack that reconstructs stimuli using conditioned diffusion models or autoregressive transformers with brain-signal serialization; and a "Unified" Stack of brain foundation models that integrate spatio-temporal transformers, cross-modal knowledge distillation, and parameter-efficient fine-tuning (LoRA/PEFT) for generalizable, multi-subject decoding.

Figure: Core AI Technology Stack System for Brain Decoding — covering the Encoder Stack (neural representation & cross-modal alignment), Decoder Stack (generative decoding & reconstruction), and the Unified Stack (brain foundation models & multimodal fusion).

🖼️ Visual Reconstruction

📂 fMRI → Image (click to expand)

Year	Title	Arch	Feature	Links
2023	High-Resolution Image Reconstruction with Latent Diffusion Models from Human Brain Activity	Diffusion	Direct fMRI-to-LDM mapping without fine-tuning	[Code]
2023	Seeing Beyond the Brain: MinD-Vis	Diffusion	Large-scale resting-state fMRI pre-training + sparse coding	[Code]
2023	Brain-Diffuser: Natural Scene Reconstruction from fMRI Signals	Diffusion	VDVAE low-level + CLIP/BLIP high-level dual-pipeline reconstruction	[Code]
2023	Reconstructing the Mind's Eye: MindEye	Diffusion	Dual-path: contrastive retrieval + diffusion prior	[Code]
2023	UniBrain: Unify Image Reconstruction and Captioning from Human Brain Activity	Diffusion	First unified fMRI-to-image + captioning in a single LDM model
2024	MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data	Diffusion	Cross-subject transfer via functional alignment; only 1hr data needed	[Code] [Website]
2024	MindBridge: A Cross-Subject Brain Decoding Framework	Diffusion	Single model for multi-subject; bio-inspired aggregation	[Code]

📂 EEG → Image (click to expand)

Year	Title	Arch	Feature	Links
2024	DreamDiffusion: Generating High-Quality Images from Brain EEG Signals	Diffusion	Temporal masking pre-train + CLIP alignment; first EEG-to-image	[Code]
2024	Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (ATM)	Diffusion	Adaptive Thinking Mapper (ATM) encoder; zero-shot cross-subject reconstruction	[Code]

📂 EEG → Video (click to expand)

Year	Title	Arch	Feature	Links
2024	EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals	Diffusion	Seq2Seq EEG encoder; first EEG-to-video reconstruction; 79.8% semantic accuracy	[Code]

📂 fMRI → Video (click to expand)

Year	Title	Arch	Feature	Links
2023	Cinematic Mindscapes: High-Quality Video Reconstruction from Brain Activity	Diffusion	Spatiotemporal attention + contrastive learning; arbitrary frame-rate	[Code] [Website]
2025	Animate Your Thoughts: Reconstruction of Dynamic Natural Vision from Human Brain Activity	Diffusion	Decouple fMRI signals into semantic, structural, and motion features, then decode them to each frame of synthesized GIFs	[Code]

---

🗣️ Speech & Language Decoding

📂 Invasive Speech — ECoG / Intracortical (click to expand)

Year	Title	Arch	Feature	Links
2023	A High-Performance Speech Neuroprosthesis	RNN	62 wpm; first large-vocab decoding (125k words)	[Code]
2023	A High-Performance Neuroprosthesis for Speech Decoding and Avatar Control	RNN	Real-time avatar control with facial expression + speech
2025	A Streaming Brain-to-Voice Neuroprosthesis	RNN-Transducer	80ms streaming decoding; real-time speech synthesis	[Code]

📂 Non-invasive Semantic — fMRI / EEG / MEG (click to expand)

Year	Title	Arch	Feature	Links
2023	Semantic Reconstruction of Continuous Language from Non-invasive Brain Recordings	Transformer	GPT autoregressive decoding + beam search; multi-cortex support
2024	DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation	Transformer	Discrete codebook alignment to LLM; no word-level gaze annotation
2025	Decoding Individual Words from Non-invasive Brain Recordings (EEG/MEG)	Transformer	Deep learning pipeline decoding individual words from EEG and MEG signals

---

🎯 Motor & Intention Decoding

📂 Motor Imagery Papers (click to expand)

Year	Title	Arch	Feature	Links
2024	CTNet: A Convolutional Transformer Network for EEG-based Motor Imagery Classification	CNN-Transformer	CNN local features + Transformer global dependencies
2023	EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization	Conformer	Conv + self-attention; CAM-based topographic visualization	[Code]
2022	ATCNet: Attention Temporal Convolutional Network for EEG-based Motor Imagery Classification	TCN	Sliding window + multi-head attention + TCN residual	[Code]

---

🧩 Brain Foundation Models

📂 Foundation Model Papers (click to expand)

Year	Title	Arch	Feature	Links
2024	LaBraM: Large Brain Model for Learning Generic Representations with Tremendous EEG Data	Transformer	Pre-trained on ~2,500 hours EEG across 20+ datasets; vector-quantized neural spectrum prediction	[Code]
2023	BRANT: Foundation Model for Intracortical Neural Signal	Transformer	Spatiotemporal Transformer pre-trained on large-scale intracortical data

---

📏 Metrics & Tools

Evaluation Metrics

Category	Metric	Description
Encoding	Pearson r, R²	Correlation between predicted and actual brain activity
Low-level Reconstruction	PixCorr, SSIM, PSNR	Pixel-level similarity
High-level Reconstruction	CLIP Score, Inception Score	Semantic/perceptual similarity
Classification	Accuracy, F1, AUC	Standard classification metrics
Retrieval	Top-k Accuracy, MRR	Retrieval-based evaluation

Software & Libraries

Tool	Description	Link
MNE-Python	MEG, EEG, sEEG, ECoG, NIRS analysis	[Website] [GitHub]
Nilearn	Statistical learning on fMRI data	[Website] [GitHub]
Braindecode	Deep learning for EEG/ECoG/MEG decoding; EEGNet, ShallowNet, etc.	[Website] [GitHub]
TorchEEG	PyTorch library for EEG processing & models	[GitHub]
Net2Brain	Compare DNN activations with brain activity (RSA, encoding)	[GitHub]
Neural_Decoding	Classic + DL decoders (Kalman, Wiener, LSTM, etc.)	[GitHub]
PyCortex	fMRI visualization on cortical surface	[GitHub]
RSA Toolbox	Representational Similarity Analysis	[GitHub]

Benchmark Platforms

Platform	Description	Link
Algonauts Project	Annual challenge for predicting brain responses to visual stimuli	[Website]
Brain-Score	Benchmark for comparing DNNs with primate visual cortex	[Website] [GitHub]
MOABB	Mother of All BCI Benchmarks; 36 EEG datasets, 30 pipelines	[Website] [GitHub]

---

🏥 Clinical Application Cases

Recent breakthroughs demonstrating real-world clinical impact.

Case	Year	Description	Links
Synchron & Apple: Thought-Controlled iPad	2025	ALS patient controlled iPad via Stentrode implant + Apple BCI HID protocol—navigating apps, composing texts using only thoughts	[News] [Video]
Neuralink Telepathy	2024	First Neuralink human implant; quadriplegic patient played chess & Civilization VI via cursor control using thoughts alone	[News] [Video]
UC Davis ALS Speech BCI	2024	Restored speech for ALS patient with >97% accuracy; preserved voice identity using high-density ECoG	[Press] [Paper]

---

📚 Learning Resources

📺 Video Tutorials & Courses

📂 Video Tutorials (click to expand)

Resource	Description	Link
Neuromatch Academy	World-class open course on computational neuroscience; encoding/decoding basics	[Website] [YouTube] [Bilibili]
INCF: Deep Learning in Neuroscience	Beginner-level DL for neuroscience applications	[Website]

📖 Textbooks & Reading

📂 Textbooks & Reading (click to expand)

Resource	Description	Link
Deep Learning (Goodfellow et al.)	Deep learning bible; free online	[Website]
Awesome-Brain-Encoding-Decoding	Curated paper list	[GitHub]

🌐 Communities

📂 Communities (click to expand)

Community	Description	Link
NeuroAI WeChat Group	Chinese community for brain + AI research	Contact via WeChat number MobiusAI
BCI Society	International BCI research community	[Website]
OHBM	Organization for Human Brain Mapping	[Website]

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. See CONTRIBUTING.md for guidelines.

---

If you find this guide helpful, please consider giving it a ⭐!