Skip to content

Expose all vLLM CLI arguments #13

@iPieter

Description

@iPieter

Goal
llmq run worker <model-name> <queue-name> should accept all vllm cli arguments, like --dtype or --max-model-len.

Full list is here: https://docs.vllm.ai/en/latest/cli/serve.html#modelconfig

Considerations

  • Perhaps we want only the modelConfig?

Sub-issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions