Awesome-Weight-Space-Learning

This repo will be continuously updated. Don't forget to star it and keep tuned!

Weight Space Learning

Weight Space Learning is a research perspective that shifts focus from studying neural networks only through their input–output functions to directly analyzing and leveraging their parameters. Unlike conventional training, which treats weights merely as optimization variables, weight space learning regards them as a meaningful domain of study and operation. Existing works in this area can be organized along three complementary dimensions: (1) weight space understanding, which investigates the geometry, symmetry, and statistical properties of weights; (2) weight space discrimination, which treats weights as a modality for tasks such as embedding, retrieval, and behavior prediction; and (3) weight space generation, which explores how new parameters can be produced via generative models, hypernetworks, or model merging. This framing highlights weight space learning as distinct from function-space or purely optimization-centric views, aiming to build a systematic foundation for reasoning about, operating on, and reusing neural network parameters.

Table of Contents

• Awesome-Weight-Space-Learning
  • Weight Space Learning
  • Table of Contents
  • Weight Space Understanding
    • Symmetry
    • Initialization
  • Weight Space Discrimination
    • Representation
    • Retrieval
    • Behavior Prediction
  • Weight Space Generation
    • Generative Model
    • HyperNet
    • Merging
  • Others
    • Model Zoo
    • Survey
    • Thesis

Weight Space Understanding

Symmetry

• [ICML 23] Equivariant Architectures for Learning in Deep Weight Spaces [PDF] [Code]
• [NeurIPS 23] Permutation Equivariant Neural Functionals [PDF] [Code]
• [NeurIPS 23] Neural Functional Transformers [PDF] [Code]
• [ICML 24] Equivariant Deep Weight Space Alignment [PDF] [Code]
• [NeurIPS-NeurReps 23] Data Augmentations in Deep Weight Spaces [PDF]
• [NeurIPS 24] Universal Neural Functionals [PDF] [Code]
• [ICML 24] Learning Useful Representations of Recurrent Neural Network Weight Matrices [PDF]
• [NeurIPS 24] Graph Neural Networks for Learning Equivariant Representations of Neural Networks [PDF] [Code]
• [NeurIPS 24] The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof [PDF] [Code]
• [NeurIPS 24] Scale Equivariant Graph Metanetworks [PDF] [Code]
• [NeurIPS 24] Monomial Matrix Group Equivariant Neural Functional Networks [PDF] [Code]
• [ICLR 25] Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representations [PDF] [Code]
• [ICML 25] Equivariant Polynomial Functional Networks [PDF] [Code]
• [ICLR 25] Equivariant Neural Functional Networks for Transformers [PDF] [Code]
• [ICML 25] Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion [PDF] [Code]
• Parameter Symmetry Potentially Unifies Deep Learning Theory [PDF]
• [CVPR 25] End-to-End Implicit Neural Representations for Classification [PDF] [Code]
• [ICML-HiLD 25] Symmetries in Weight Space Learning: To Retain or Remove? [PDF]
• GradMetaNet: An Equivariant Architecture for Learning on Gradients [PDF]

Initialization

• [ICML 23] Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models? [PDF] [Code]
• [NeurIPS 23] Sampling weights of deep neural networks [PDF] [Code]
• [CVPR 25] WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models [PDF] [Code]
• [ECCV 24] Efficient Training with Denoised Neural Weights [PDF] [Code]
• [ICLR 25] RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks [PDF] [Code]

Weight Space Discrimination

Representation

• [ICML-TAGML 23] Neural Networks Are Graphs! Graph Neural Networks for Equivariant Processing of Neural Networks [PDF]
• [NeurIPS 24] Set-based Neural Network Encoding Without Weight Tying [PDF]
• [ICLR 24] Graph Metanetworks for Processing Diverse Neural Architectures [PDF]
• [ICML 24] Improved Generalization of Weight Space Networks via Augmentations [PDF] [Code]
• Dynamic Neural Graph: Facilitating Temporal Dynamics Learning in Deep Weight Space [PDF] [Code]
• Structure and Behavior in Weight Space Representation Learning [PDF]
• [ECCV 24] Neural Metamorphosis [PDF] [Code]
• [CVPR 25] Learning on Model Weights using Tree Experts [PDF] [Code]
• Deep Linear Probe Generators for Weight Space Learning [PDF]
• From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields [PDF] [Code]
• Weight Space Representation Learning on Diverse NeRF Architectures [PDF]
• [ICLR-WSL 25] Adversarial Robustness in Parameter-Space Classifiers [PDF] [Code]
• Weight-Space Linear Recurrent Neural Networks [PDF] [Code]
• GNN-based Unified Deep Learning [PDF] [Code]
• Cross-Model Semantics in Representation Learning [PDF]

Retrieval

• Can this Model Also Recognize Dogs? Zero-Shot Model Search from Weights [PDF]

Behavior Prediction

• Predicting Neural Network Accuracy from Weights [PDF] [Code]
• [NeurIPS 21] Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction [PDF] [Code]
• [ICLR-NC 21] Training and Generating Neural Networks in Compressed Weight Space [PDF] [Code]
• [ICSE-SEIP 23] Runtime Performance Prediction for Deep Learning Models with Graph Neural Network [PDF]
• [ICML 24] Recovering the Pre-Fine-Tuning Weights of Generative Models [PDF] [Code]
• Dataset Size Recovery from LoRA Weights [PDF]
• Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization [PDF]
• Towards Meta-Models for Automated Interpretability [PDF]
• Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models [PDF]
• [Computer Networks] Model Parameter Prediction Method for Accelerating Distributed DNN Training [PDF]

Weight Space Generation

Generative Model

• [ICML-PT 22] Hyper-Representations for Pre-Training and Transfer Learning [PDF]
• Learning to Learn with Generative Models of Neural Network Checkpoints [PDF] [Code]
• [NeurIPS 22] Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights [PDF] [Code]
• [TMLR 23] Meta-Learning via Classifier(-free) Diffusion Guidance [PDF]
• [ICML 23] Learning to Boost Training by Periodic Nowcasting Near Future Weights [PDF] [Code]
• [ICCV 23] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion [PDF] [Code]
• [AAAI 24] MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning [PDF]
• [ICML 25] WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models [PDF] [Code]
• [ICLR 25] Diffusion-Based Neural Network Weights Generation [PDF] [Code]
• Neural Network Diffusion [PDF] [Code]
• BEND: Bagging Deep Learning Training Based on Efficient Neural Network Diffusion [PDF]
• [CIKM 24] Beyond Aggregation: Efficient Federated Model Consolidation with Heterogeneity-Adaptive Weights Diffusion [PDF]
• [ICML 24] Towards Scalable and Versatile Weight Space Learning [PDF] [Code]
• DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion [PDF]
• Conditional LoRA Parameter Generation [PDF] [Code]
• [NeurIPS 24] Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments [PDF] [Code]
• [ICLR 25] Accelerating Training with Neuron Interaction and Nowcasting Networks [PDF] [Code]
• [AAAI 25] pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning [PDF]
• Generating GFlowNets as You Wish with Diffusion Process [PDF]- Generating GFlowNets as You Wish with Diffusion Process [PDF]
• [Recsys 25] Paragon: Parameter Generation for Controllable Multi-Task Recommendation [PDF] [Code]
• [ICLR 25] Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning [PDF] [Code]
• [CVPR 25] Few-shot Implicit Function Generation via Equivariance [PDF] [Code]
• [IJCAI 25] In-Context Meta LoRA Generation [Code]
• Recurrent Diffusion for Large-Scale Parameter Generation [PDF] [Code]
• Learning to Learn Weight Generation via Local Consistency Diffusion [PDF]
• ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion [PDF]
• [ICLR-WSL 25] Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction [PDF] [Code]
• [ICLR-WSL 25] Instruction-Guided Autoregressive Neural Network Parameter Generation [PDF]
• NeuroGen: Neural Network Parameter Generation via Large Language Models [PDF]
• Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarios [PDF]
• Generative Modeling of Weights: Generalization or Memorization? [PDF] [Code]
• [ICML-EXAIT 25] Reimagining Parameter Space Exploration with Diffusion Models [PDF]
• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights [PDF] [Code]
• Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting [PDF]

HyperNet

• [ICML 25] Learngene Tells You How to Customize: Task-Aware Parameter Initialization at Flexible Scales [PDF] [Code]
• [NeurIPS-AdvML 24] Learning to Forget using Hypernetworks [PDF]
• [CVPR 25] HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories [PDF]
• LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization [PDF]

Merging

• [NeurIPS 20] Meta-Consolidation for Continual Learning [PDF] [Code]
• [CVPR 25] PLeaS — Merging Models with Permutations and Least Squares [PDF] [Code]

Others

Model Zoo

• [NeurIPS 22] Model Zoos: A Dataset of Diverse Populations of Neural Network Models [PDF] [Code]
• [ICLR-SNN 23] Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models [PDF] [Code]
• [EDBT 25] Model Lakes [PDF]
• [ICLR 25] Unsupervised Model Tree Heritage Recovery [PDF] [Code]
• [NeurIPS 24] Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes [PDF] [Code]
• [NeurIPS 24] Interpreting the Weight Space of Customized Diffusion Models [PDF] [Code]
• Model Zoos for Benchmarking Phase Transitions in Neural Networks [PDF]
• We Should Chart an Atlas of All the World's Models [PDF] [Code]
• Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training [PDF] [Code]
• [ICLR-WSL 25] A Model Zoo of Vision Transformers [PDF] [Code]
• [Electronics] An Open Dataset of Neural Networks for Hypernetwork Research [PDF]
• [ICCS 25] Towards Weight-Space Interpretation of Low-Rank Adapters for Diffusion Models [PDF]

Survey

• A Brief Review of Hypernetworks in Deep Learning [PDF]
• Implicit Neural Representation in Medical Imaging: A Comparative Survey [PDF]
• Learning from Models Beyond Fine-Tuning [PDF]
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities [PDF]
• Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey [PDF]
• Symmetry in Neural Network Parameter Spaces [PDF]

Thesis

• [PhD Thesis] Hyper-Representations: Learning from Populations of Neural Networks [PDF]
• [PhD Thesis] Acquiring and Adapting Priors for Novel Tasks via Neural Meta-Architectures [PDF]
• [MSc Thesis] Geometric Flow Models over Neural Network Weights [PDF] [Code]