Created an api inference script with its supporting documentation#959
Conversation
albertodepaola
left a comment
albertodepaola
left a comment
There was a problem hiding this comment.
Thanks for putting this together! Overall looks good, added a couple of comments on the args to the script.
| if api_key is not None: | ||
| os.environ["LLAMA_API_KEY"] = api_key | ||
| else: | ||
| env_var_name = f"{args.provider.upper()}_API_KEY" |
There was a problem hiding this comment.
The provider param seems to be missing from the parser and the docs. We should document how this can be used to change the env variable lookup. Having said that, these other providers might not be compatible with LlamaAPIClient. The options would be to use OpenAI compatible layer in Llama API (https://llama.developer.meta.com/docs/features/compatibility/?team_id=465615129850692) or remove this alternative.
There was a problem hiding this comment.
Thanks, @albertodepaola for noticing that! I have added the provider argument back. I have only included Llama and OpenAI. Please let me know what you think!
| ) | ||
| args = parser.parse_args() | ||
|
|
||
| api_key: Optional[str] = args.api_key |
There was a problem hiding this comment.
nit: add the import to the top of the file.
There was a problem hiding this comment.
Gotcha! I added info about that in the README
| - Passed the API key using the `--api-key` argument | ||
| - Set the appropriate environment variable | ||
|
|
||
| ### Known Issues |
There was a problem hiding this comment.
When fixing the comments below, this will become irrelevant, remove or update accordingly!
albertodepaola
left a comment
albertodepaola
left a comment
There was a problem hiding this comment.
LG, thanks for addressing the comments!
3 participants
Created an API inference script where users could pick a llama model, set their api keys, prompt the model and see the results on a Gradio UI.
Tested the system and attached a snap shot of the UI in action.
