Skip to content

Responses vLLM backend passes TP environment value as a string #275

Description

@sylvesterkaczmarek

Summary

The Responses API vLLM backend defines tensor parallelism with:

TP = os.environ.get("TP", 2)

When TP is unset, the fallback is integer 2. When a user configures the environment variable, os.environ.get() returns a string, and that string is passed directly to:

LLM(..., tensor_parallel_size=TP, ...)

tensor_parallel_size is an integer configuration value, so the explicit environment override changes the value's type instead of only changing its size.

Impact

The documented/configurable TP path can fail during vLLM engine setup when users set values such as TP=4, even though the default path works.

Proposed resolution

Parse the environment variable as an integer:

TP = int(os.environ.get("TP", 2))

Add a regression that imports the backend with TP=4 and verifies load_model() passes integer 4 to LLM.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions