api inference mini fork #109
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oOraph
force-pushed
the
dev/api-inference-mini-fork
branch
5 times, most recently
from
b93b802 to
7f17bb6
Compare
oOraph
commented
May 5, 2025
| if default_num_steps: | ||
| kwargs["num_inference_steps"] = int(default_num_steps) | ||
|
|
||
| if "guidance_scale" not in kwargs: |
There was a problem hiding this comment.
useful for sd 3.5 turbo -> we want guidance scale 0 by default (e.g when not specified by user) because the num steps is too low, so that generated images are ok
oOraph
commented
May 5, 2025
| @@ -20,3 +23,7 @@ def strtobool(val: str) -> bool: | |||
| raise ValueError( | |||
| f"Invalid truth value, it should be a string but {val} was provided instead." | |||
| ) | |||
|
|
|||
|
|
|||
| def api_inference_compat(): | |||
There was a problem hiding this comment.
with this env var we intend to handle the small response differences between the api inference widgets on the hub and on endpoints ui. TODO: we should probably unify both widgets instead
oOraph
commented
May 5, 2025
| Route("/predict", predict, methods=["POST"]), | ||
| Route("/metrics", metrics, methods=["GET"]), | ||
| ] | ||
| if api_inference_compat(): |
There was a problem hiding this comment.
I only activated multi task for api inference (as a test) but we may want to remove this condition and just always support it if we're satisified with it)
There was a problem hiding this comment.
actually thinking about it -> we may want a separate env var (and keep deactivated by default for regular users, and provide an option for it in endpoints instead) because the pod may consume more ram than expected (due to the pipeline duplications) with this route
oOraph
force-pushed
the
dev/api-inference-mini-fork
branch
5 times, most recently
from
459f3b6 to
39db7c6
Compare
oOraph
force-pushed
the
dev/api-inference-mini-fork
branch
from
c71a4c5 to
c5565c2
Compare
oOraph
force-pushed
the
dev/api-inference-mini-fork
branch
3 times, most recently
from
67491ac to
0818705
Compare
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
…cale defaults to 0 when num steps <=4 Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
More flexibility than an exact string match since there can be some additional params Signed-off-by: Raphael Glon <[email protected]>
…ard compat Signed-off-by: Raphael Glon <[email protected]>
…ion for sentence transformers Signed-off-by: Raphael Glon <[email protected]>
no reason not to accept it Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
…tch size dim Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
return an error instead Signed-off-by: Raphael Glon <[email protected]>
… we do not know what to do with Signed-off-by: Raphael Glon <[email protected]>
backported and adapted from https://github.com/huggingface/api-inference-community/blob/main/docker_images/diffusers/app/idle.py 1. adding gunicorn instead of uvicorn to allow for wsgi/asgi workers to easily be suppressed when idle whithout stopping the entire service -> easy way to release memory whithout digging into the depth of the imported modules 2. memory consuming libs lazy load (transformers, diffusers, sentence_transformers) 3. pipeline lazy load as well The first 'cold start' request tends to be a bit slower than others but the footprint is reduced to the minimum when idle Signed-off-by: Raphael Glon <[email protected]>
To optimize build time and enhance layer reuse Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
Signed-off-by: Raphael Glon <[email protected]>
oOraph
force-pushed
the
dev/api-inference-mini-fork
branch
from
1ea58f3 to
2eda42a
Compare
Signed-off-by: Raphael Glon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.