Describe the bug
In the README and when running help, there are references to openai endpoint support, but it appears to no longer exist.
$ lighteval endpoint --help
Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...
Evaluate models using some endpoint (tgi, inference endpoint, openai) as backend.
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Evaluation Backends ────────────────────────────────────────────────────────────────────────────────╮
│ inference-endpoint Evaluate models using inference-endpoints as backend. │
│ tgi Evaluate models using TGI as backend. │
│ litellm Evaluate models using LiteLLM as backend. │
│ inference-providers Evaluate models using LiteLLM as backend. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
$ lighteval endpoint openai --help
Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...
Try 'lighteval endpoint --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such command 'openai'. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
Expected behavior
If the openai/openai-compatible support was removed, please remove it from your documentation or indicate clearly that it is no longer supported.
Describe the bug
In the README and when running help, there are references to openai endpoint support, but it appears to no longer exist.
Expected behavior
If the openai/openai-compatible support was removed, please remove it from your documentation or indicate clearly that it is no longer supported.