Selectively choose which parameters to offload

[taehyunzzz]

Is there a way to select which parameters I want to offload?

For example I want to offload only the big model parameters like FFN to CPU memory and fetch them when needed,
whereas all the other layers such as convolution stays in the GPU for good.

Does the current deepspeed allow such implementation?
If not, has anybody seen a custom user-implementation of such thing?
Where should I mainly modify to achieve this?

[tjruwase]

Try "stage3_param_persistence_threshold" to ensure that parameters smaller than a threshold are not partitioned or offloaded.

See here for details.

[taehyunzzz]

@tjruwase Thank you. I have another question regarding ZeRO-inference blog post.
They mentioned pinning some parameters to the GPU memory, rather than allowing swap between GPU and CPU.
Does the current deepspeed API provide such parameter pinning functionality?

[ethansmith2000]

are there alternative methods, say, if i wanted to offload non-trainable parameters?

[tjruwase]

@ethansmith2000, there are no methods for offloading only non-trainable parameters.

[tjruwase]

Use the following: https://deepspeed.readthedocs.io/en/latest/zero3.html#deepspeed.runtime.zero.config.DeepSpeedZeroConfig.model_persistence_threshold