llama4 request

[SicariusSicariiStuff]

llama4 is huge, having the ability to somehow prune it \ separate the experts will help the community a great deal.

[yukiarimo]

+1

[davzoku]

+1, besides pruning and separating experts, the possibility of merging existing trained models into the new llama 4 architecture will be interesting too. Imagine having separate 1B models and merge them into llama 4 architecture, this will be very helpful for further experimentations.

[MrJackSpade]

I see a commit to add a definition for Llama4, however I get an error message when trying to actually merge

RuntimeError: Tensor vision_model.model.layers.33.attention.v_proj.bias required but not present in model models--meta-llama--Llama-4-Maverick-17B-128E

[davzoku]

Linking related PR Add Llama4ForConditionalGeneration by cg123 · Pull Request #552 · arcee-ai/mergekit for easy tracking