Skip to content

Introducing MoELayer and basic usage for the head network. #575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open

Introducing MoELayer and basic usage for the head network. #575

wants to merge 35 commits into from

Conversation

@Joao-L-S-Almeida
Copy link
Collaborator

@Joao-L-S-Almeida Joao-L-S-Almeida commented Apr 3, 2025 •
edited
Loading

  • Introducing the MoELayer.
  • A partial support for scalar models (classification).
  • An example for EuroSat.
  • Some basic tests.

To be merged in version 1.2

@Joao-L-S-Almeida
A basic MoE layer class
810bdf3 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Adapting the factories and peft to support MoE
0b6dc6b 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Solving issues for configuration and instantiation
d3dda75 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
The variable n_vars isnecessary to inform the number of patches in …
df29cb7 
…the transformer layer input. How to automaticalluy estimate it ?

Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida Joao-L-S-Almeida self-assigned this Apr 3, 2025
@Joao-L-S-Almeida Joao-L-S-Almeida linked an issue Apr 3, 2025 that may be closed by this pull request
MoE layers combined with LoRA (MoLoRA) #570
Closed
@Joao-L-S-Almeida Joao-L-S-Almeida changed the title [WiP] MoE for LoRA (MoLoRA) [WiP] Attempting to use MoE for LoRA (MoLoRA) Apr 3, 2025
@Joao-L-S-Almeida
Merge branch 'main' into introduce/moe
99de383
@Joao-L-S-Almeida
Reverting unnecessarychanges
8dd0758 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Reverting unnecessary changes
9672c32 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
unnecessary breaklines
65aaec9 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida Joao-L-S-Almeida changed the title [WiP] Attempting to use MoE for LoRA (MoLoRA) [WiP] Introducing MoELayer Apr 25, 2025
@Joao-L-S-Almeida
Example of EuroSat in notebook
c0b5ff9 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Merge branch 'classification/gt/int' into example/eurosat
6d2c4c4
@Joao-L-S-Almeida
Example of EuroSat in notebook
6b0d7b2 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Updating notebook
ec5b552 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
EuroSat as script
8bc1864 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Clear cells
6c45668 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Merge branch 'example/eurosat' into introduce/moe
153e36f
@Joao-L-S-Almeida
Merge branch 'introduce/moe' of github.com:IBM/terratorch into introd…
27ae391 
…uce/moe
@Joao-L-S-Almeida
Some inputs could require a reshaping in order to use the dense gate …
9970293 
…network

Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
A special case for a MoE head in scalar models
c598a0c 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
tests for MoELayer
f9762c6 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
MoE for the eurosat example
e6aff02 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Adjusting dimensions for weights and outputs in MoE
1d7846c 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Some basic tests for MoE
1e7f056 
Signed-off-by: Joao Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Updating EuroSat example to work with MoE
a43f379 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
The default value for n_vars must be 1
d1c2243 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida Joao-L-S-Almeida mentioned this pull request Apr 30, 2025
Closed
@Joao-L-S-Almeida Joao-L-S-Almeida changed the title [WiP] Introducing MoELayer [WiP] Introducing MoELayer and basic usage for the head network. May 12, 2025
@Joao-L-S-Almeida
The input of the gating network wasn't in the same device of the weights
2fdd707 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>

Completig tests for IdentityDecoder

Signed-off-by: João Lucas de Sousa Almeida <[email protected]>

A method to unsqueeze the gating weights when necessary

Signed-off-by: João Lucas de Sousa Almeida <[email protected]>

MoE head layer for PixelWise models

Signed-off-by: João Lucas de Sousa Almeida <[email protected]>

Tests for PixelWise Models + MoE head layer

Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida
Fixing assertion for segmentation tests
e8af833 
Signed-off-by: João Lucas de Sousa Almeida <[email protected]>

Annotations for MoELayer

Signed-off-by: João Lucas de Sousa Almeida <[email protected]>
@Joao-L-S-Almeida Joao-L-S-Almeida changed the title [WiP] Introducing MoELayer and basic usage for the head network. Introducing MoELayer and basic usage for the head network. May 12, 2025
@Joao-L-S-Almeida
Copy link
Collaborator Author

@paolo-fraccaro @blumenstiel @romeokienzler Is it something we want to have for 1.1 ?

@Joao-L-S-Almeida Joao-L-S-Almeida added 1.2 and removed 1.1 labels May 28, 2025
@Joao-L-S-Almeida Joao-L-S-Almeida removed the request for review from paolofraccaro August 26, 2025 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blumenstiel blumenstiel Awaiting requested review from blumenstiel

@romeokienzler romeokienzler Awaiting requested review from romeokienzler

At least 1 approving review is required to merge this pull request.

Assignees

@Joao-L-S-Almeida Joao-L-S-Almeida

Labels

1.2 experimental

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

MoE layers combined with LoRA (MoLoRA)

2 participants

@Joao-L-S-Almeida