-
Notifications
You must be signed in to change notification settings - Fork 26
DeepSpeed and FairScale #78
Copy link
Copy link
Open
Labels
P3Nice to have, not working on it for nowNice to have, not working on it for nowenhancementNew feature or requestNew feature or requestperformancePerformance issues: resource usage, scaling, GPU efficiencyPerformance issues: resource usage, scaling, GPU efficiencytrainingIssues with training loop, optimization, GPU usageIssues with training loop, optimization, GPU usage
Description
Metadata
Metadata
Assignees
Labels
P3Nice to have, not working on it for nowNice to have, not working on it for nowenhancementNew feature or requestNew feature or requestperformancePerformance issues: resource usage, scaling, GPU efficiencyPerformance issues: resource usage, scaling, GPU efficiencytrainingIssues with training loop, optimization, GPU usageIssues with training loop, optimization, GPU usage
Type
Fields
Give feedbackNo fields configured for issues without a type.
Huggingface recently started support for DeepSpeed and FairScale which supposedly give major speedups in both multi-gpu and single-gpu settings. The feature is still experimental but worthwhile to look into and see if we can get any performance boosts.
Documentation