Please try to add support for pruning bigger ZAYA models.
Expected results:
- Pruned student scores better than 8B after distillation from its 74B teacher.
- Less GPU RAM needed to get scores near bigger ZAYA https://huggingface.co/Zyphra/ZAYA1-74B-preview
- Distill process runs faster than from Qwen-4B dense, since only 4B active of 74B MoE.
Please try to add support for pruning bigger ZAYA models.
Expected results: