Skip to content

Feature Request: Support extra_body Parameter for Advanced Model Features #898

Open
@wqj97

Description

@wqj97

Background
vLLM currently supports various model features through configuration parameters, but lacks support for passing additional model-specific parameters through extra_body, which is particularly important for features like structured output.
https://github.com/vllm-project/vllm/blob/v0.6.0/vllm/engine/arg_utils.py#L276

Current OpenAI implementation

completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Generate a user profile"}],
    extra_body={
        "guided_json": Test.schema_json,
        "guided_decoding_backend": "lm-format-enforcer"
    }
)

Proposed implementation

resp, err := integrations.LLMClient.Client.CreateChatCompletion(
		ctx,
		openai.ChatCompletionRequest{
			Model: "...",
			Messages: []openai.ChatCompletionMessage{
				...
			},
			ExtraBody: map[string]any{
				....
			},
		},
	)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions