Motivation
The Qwen NLP adapter currently does not use structured outputs, which may result in errors during inference. Alibaba Cloud has recently launched its own structured outputs feature, which requires supported Qwen models to adhere to provided Pydantic schemas.
Solution Proposal
Modify the Qwen NLP adapter to use the JSON Schema mode as described here for supported models, with JSON Object mode being retained as a fallback in other instances. There are currently three supported Qwen models, with Structured Outputs being available only in the China (domestic) region. This means that implementing #698 would facilitate easier fallback for most users.
Discussion
This FR is paired with #678, which would provide the same constrained sampling feature for user-provided open weights models.
Motivation
The Qwen NLP adapter currently does not use structured outputs, which may result in errors during inference. Alibaba Cloud has recently launched its own structured outputs feature, which requires supported Qwen models to adhere to provided Pydantic schemas.
Solution Proposal
Modify the Qwen NLP adapter to use the JSON Schema mode as described here for supported models, with JSON Object mode being retained as a fallback in other instances. There are currently three supported Qwen models, with Structured Outputs being available only in the China (domestic) region. This means that implementing #698 would facilitate easier fallback for most users.
Discussion
This FR is paired with #678, which would provide the same constrained sampling feature for user-provided open weights models.