Use fallback options to guarantee a successful output

[moritz-gross]

When I first looked into this project, I thought you could get a guarantee (not just reasonable high chance) that you will get a response in the required schema.
From my experience, this isn't the case, unless I used the library incorrectly.

I think this could be implemented by using a fallback in the stream, i.e. when you receive a token that is incompatible with the schema, you instead go on with the 2nd most likely option, etc.

How feasible is this idea? For GPT-4, you have access to the logprobs, which would enable this approach, right?

[moritz-gross]

An example where llama3.1 failed to return a valid response (which worked for gpt-4o though):

code

from typing import List

import instructor
from openai import OpenAI
from pydantic import BaseModel


class City(BaseModel):
    name: str
    country: str


class Cities(BaseModel):
    cities: List[City]


# enables 'response_model' in create call
client = instructor.from_openai(OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required, but unused
),
    mode=instructor.Mode.JSON,
)

city_list = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {
            "role": "user",
            "content": "List 5 cities from around the world and their countries"
        },
    ],
    response_model=Cities,
)

print(city_list)

error stack trace

/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/bin/python /Users/moritzgross/PycharmProjects/InstructorTesting/main.py 
Traceback (most recent call last):
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/retry.py", line 193, in retry_sync
    raise e
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/retry.py", line 164, in retry_sync
    return process_response(
           ^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/process_response.py", line 144, in process_response
    model = response_model.from_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/function_calls.py", line 152, in from_response
    return cls.parse_json(completion, validation_context, strict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/function_calls.py", line 340, in parse_json
    return cls.model_validate_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/pydantic/main.py", line 597, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 5 validation errors for Cities
cities.1.country
  Field required [type=missing, input_value={'name': 'City ', 'title': 'Title'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
cities.2.name
  Field required [type=missing, input_value={'$defer': {'_id': 'city'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
cities.2.country
  Field required [type=missing, input_value={'$defer': {'_id': 'city'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
cities.3.name
  Field required [type=missing, input_value={'valuename': {'name': 'V...$ref': "'#/$defs/City}"}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
cities.3.country
  Field required [type=missing, input_value={'valuename': {'name': 'V...$ref': "'#/$defs/City}"}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/retry.py", line 158, in retry_sync
    for attempt in max_retries:
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 443, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 419, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x1072976b0 state=finished raised ValidationError>]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/main.py", line 25, in <module>
    city_list = client.chat.completions.create(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/client.py", line 116, in create
    return self.create_fn(
           ^^^^^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/patch.py", line 143, in new_create_sync
    response = retry_sync(
               ^^^^^^^^^^^
  File "/Users/moritzgross/PycharmProjects/InstructorTesting/.venv/lib/python3.12/site-packages/instructor/retry.py", line 195, in retry_sync
    raise InstructorRetryException(
instructor.exceptions.InstructorRetryException: RetryError[<Future at 0x1072976b0 state=finished raised ValidationError>]

Process finished with exit code 1

Did I miss any settings to get this to work? I feel like this example should be manageable.

[gferioli0418]

Also experiencing this issue and seems to happen at random? My assumption is it has to do with how Instructor forms there prompts and if there are any retries involved.

[ivanleomk]

Hey man, if you're using the OpenAI compatible client, we don't do anything with the prompts, all we do is convert your schema to a OpenAI Compatible Schema and from there send it together with the request. We do append an error message with the generated content when a validation error is detected though.

The validation errors itself seems to be thrown because the model isn't repairing what was originally generated in a prior request. You can experiment with higher retries and see if that works!

Out of curiosity, what model are you using with this?

[moritz-gross]

llama3.1
I included the code in the details

I'm thinking of something like tracking back the current token if the current output can't be extended to match the format, and then backing off to the 2nd most likely token, then 3rd etc.

If I understand it correctly, llama.cpp already supports this? (even more general as you can supply any BFN grammar)

[gferioli0418]

We are using gpt4o model from OpenAI. Will try the higher retries method

…

On Sat, Aug 31, 2024 at 3:05 AM Ivan Leo ***@***.***> wrote: Hey man, if you're using the OpenAI compatible client, we don't do anything with the prompts, all we do is convert your schema to a OpenAI Compatible Schema and from there send it together with the request. We do append an error message with the generated content when a validation error is detected though. The validation errors itself seems to be thrown because the model isn't repairing what was originally generated in a prior request. You can experiment with higher retries and see if that works! Out of curiosity, what model are you using with this? — Reply to this email directly, view it on GitHub <#952 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOG52IDGLRFSYMFBNLYLESTZUFTLPBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE3DKMZVHA4TCMBSQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGQ4DKNBZGM4TGNFHORZGSZ3HMVZKMY3SMVQXIZI> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .