Idea: Use constrained decoding (structured output) so that you always get back the same 10 items you put in

[danielbodart]

Instead of needing to retry and incure higher costs, you can enforce the model to always return the items you have asked it to rank. The process is called constrained decoding and OpenAI supports a subset of the capability via structured outputs and JSON Schema.

Here is an example JSON Schema that says the response is going to be a JSON Array with 10 unique items in it

{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "array",
"minItems": 10,
"maxItems": 10,
"uniqueItems": true,
"items": {
"type": "string",
"enum": ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]
}
}

So you could just change the minItems, maxItems and enum to match your input.

According to the OpenAI docs there is a latency delay as it converts the schema into a grammar that is then injected into the model execution context, so this idea might have a downside or potentially they don't support this set of schema features.

(Good article BTW)

[noperator]

I'm currently maintaining Raink over at https://github.com/noperator/raink (updates will occasionally flow upstream to this source as this repo's maintainers have time).

Thanks for the idea. Will report back when if/when I give it a try.