Skip to content

feat: add post_training RuntimeConfig #2036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

cdoern
Copy link
Contributor

@cdoern cdoern commented Apr 26, 2025

What does this PR do?

certain APIs require a bunch of runtime arguments per-provider. The best way currently to pass these arguments in is via the provider config. This is tricky because it requires a provider to be pre-configured with certain arguments that a client side user should be able to pass in at runtime

Especially with the advent of out-of-tree providers, it would be great for a generic RuntimeConfig class to allow for providers to add and validate their own runtime arguments for things like supervised_fine_tune

For example: https://github.com/opendatahub-io/llama-stack-provider-kft has things like input-pvc, model-path, etc in the Provider Config. This is not sustainable nor is adding each and every field needed to the post_training API spec. RuntimeConfig has a sub-class called Config which allows for extra fields to arbitrarily be specified. It is the providers job to create its own class based on this one and add valid options, parse them, etc

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Apr 26, 2025
certain APIs require a bunch of runtime arguments per-provider. The best way currently to pass these arguments in is via the
provider config. This is tricky because it requires a provider to be pre-configured with certain arguments that a client side user should be able to pass in at runtime

Especially with the advent of out-of-tree providers, it would be great for a generic RuntimeConfig class to allow for providers to add and validate their own runtime arguments for things like supervised_fine_tune

For example: https://github.com/opendatahub-io/llama-stack-provider-kft has things like `input-pvc`, `model-path`, etc in the Provider Config.
This is not sustainable nor is adding each and every field needed to the post_training API spec. RuntimeConfig has a sub-class called Config which allows for extra fields to arbitrarily be specified. It is the providers job to create its own class based on this one and add valid options, parse them, etc

Signed-off-by: Charlie Doern <[email protected]>
@ashwinb
Copy link
Contributor

ashwinb commented Apr 26, 2025

This is tricky territory and I want to be extremely careful going down this path. At a certain point, the API will stop having meaning and only specific provider combinations will work because client code will be tied to the provider. Why have a generic API at that point? We have kept these escape hatches in the safety.run_shield() method for example, but I think that is an anti-pattern really. I would rather pull up whatever is needed and generalize those pieces and see if we can make them part of the API.

@cdoern
Copy link
Contributor Author

cdoern commented Apr 26, 2025

good point @ashwinb, maybe I could approach a larger refactor to some of the args/class structures of the post training API to better fit more generic implementations whether they be distributed or single node. Implementing some out of tree providers has led me to discover some things I think would be generally helpful, (S3 compatibility, K8s native handling at times, etc)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants