Evaluation function based on LLM grading

[msaelices]

Changes

• New is_correct() evaluation function, which asks an LLM model to return if a response is correct

Proof-of-life

[edwardmfho]

Test-ran the example cases using gpt-3.5-turbo instead of the default gpt-4 (still waiting for the API). It is a good function to be added in.

[edwardmfho]

Looks good.

[mistercrunch]

I like the idea, the only thing is the implementation is very openai-specific, which is probably fine as long as we make it clear. How about we break down evals.py into evals/__init__.py and evals/openai.py.

Goal would be to import is_correct from promptimize.evals.openai

[edwardmfho]

Should we begin with some sort of base eval function/class that could be used for other LLMs?

[lain5etf7w]

[email protected]