Skip to content

Model loading from huggingface using parallel downloads #770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 23 commits into
base: master
Choose a base branch
from
Draft

Model loading from huggingface using parallel downloads #770

wants to merge 23 commits into from
+31 −0

Conversation

dxqbYD
Copy link
Collaborator

@dxqbYD dxqbYD commented Apr 5, 2025

  • increases download speed from about 40 mbit/s to the maximum download limit of my connection, by utilizing the parallel downloads feature of the huggingface_hub API
  • works on any host including your local machine, but especially useful for cloud training, because the model is downloaded anew each training run (if you don't have persistent storage)
  • implemented only for Flux, hence the "Draft PR". Will copy to all models if you like this PR

dxqbYD and others added 20 commits January 12, 2025 16:44
@dxqbYD
override Flux transformer
a40c25e
@dxqbYD
ignore hugging face connection error for offline training
c5c08a2
@dxqbYD
pre-commit fix
b60968d
@dxqbYD
Merge branch 'Nerogar:master' into master
395d078
@dxqbYD
Merge branch 'Nerogar:master' into master
a441f04
@dxqbYD
Merge branch 'Nerogar:master' into master
0fb8512
@dxqbYD
Merge branch 'Nerogar:master' into master
a1a263d
@dxqbYD
Merge branch 'master' of https://github.com/dxqbYD/OneTrainer
ae2e7ce
@dxqbYD
clean exit
aec59b6
@dxqbYD
Merge branch 'Nerogar:master' into override_transformer
15967e6
@dxqbYD
Merge branch 'master' of https://github.com/dxqbYD/OneTrainer
93fe9e0
@dxqbYD
Merge branch 'master' of https://github.com/dxqbYD/OneTrainer
6fefde0
@dxqbYD
Merge branch 'master' of https://github.com/dxqbYD/OneTrainer
af3cea8
@dxqbYD
Merge branch 'master' of https://github.com/dxqbYD/OneTrainer
0aa6c63
@dxqbYD
Merge branch 'master' into override_transformer
7929cb3
@dxqbYD
avoid loading in float32; bugfix
bf38e55
@dxqbYD
Merge branch 'master' of https://github.com/dxqbYD/OneTrainer
3566a1d
@dxqbYD
prepare HF submodules
851c3f2
@dxqbYD
Merge branch 'override_transformer' into precache2
29182bb
@dxqbYD
merged Nerogar#639
4d0ccaa
@dxqbYD
Copy link
Collaborator Author

dxqbYD commented Apr 5, 2025

merged #639 because there is a small dependency. Find the stand-alone code of this PR here: 851c3f2

@dxqbYD
Copy link
Collaborator Author

dxqbYD commented Apr 21, 2025

merged the new master branch, which makes this PR easier to read because it was a combination before

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dxqbYD