Add CodeAct module

[TomeHirata]

This PR add a new module called CodeAct, which adopts the CodeAct architecture to manipulate the Python interpreter with predefined tools.

from dspy.predict import CodeAct
def factorial(n):
    if n == 1:
        return 1
    return n * factorial(n-1)

act = CodeAct("n->factorial", tools=[factorial])
act(n=5) # 120

[okhat]

Maybe revise to "When providing next_tool_args, the value inside the field must be in JSON format".

[okhat]

Also I'm confused about line 37, which has

                "When selecting the next_tool_name and its next_tool_args, the tool must be one of:\n",

I thought that was removed? Am I thinking of a different PR?

[TomeHirata]

Line 37 will be removed in #8190 👍

[chenmoneygithub]

Can we follow up with a code tutorial demonstrating a use case of CodeAct?

[TomeHirata]

Yup, I will file a follow up PR for documentation

[chenmoneygithub]

curious - why do we have this constraint?

[TomeHirata]

Since it is hard to define the callable object in the python interpreter. Unlike functions, we need to define class and what parameters are passed to the object initialization, which is not visible.

[chenmoneygithub]

Awesome work!

[chenmoneygithub]

should that be "besides the final Python code" or finished=True and generated_code is not None be mutual exclusive?

[TomeHirata]

Since we have an extractor at last, I designed it in a way that the code passed with finished=True is executed to minimize the interaction count.

[chenmoneygithub]

shall we move it to init, and only run this line for call-time tools? (the new change in ReAct)

[TomeHirata]

Since python interpreter is shutdown every forward, this needs to happen in forward too.

[chenmoneygithub]

We may also want to test the input flow - use a mock litellm.completion to capture the prompt, and validate our tool + code information are correctly included in the prompt.

[chenmoneygithub]

Code looks good to me! I will need to play with this new Module a bit today or tomorrow.

[chenmoneygithub]

I encountered an error when the function arg is of pydantic type:

import dspy

lm_4o_mini = dspy.LM("openai/gpt-4o-mini")
dspy.configure(
    lm=lm_4o_mini,
)


from pydantic import BaseModel

from dspy.datasets import DataLoader

kwargs = dict(fields=("claim", "supporting_facts", "hpqa_id", "num_hops"), input_keys=("claim",))
hover = DataLoader().from_huggingface(dataset_name="hover-nlp/hover", split="train", trust_remote_code=True, **kwargs)

hpqa_ids = set()
filtered_hover = []
for x in hover:
    if x["num_hops"] == 3 and x["hpqa_id"] not in hpqa_ids:
        hpqa_ids.add(x["hpqa_id"])
        filtered_hover.append(
            dspy.Example(claim=x.claim, titles=list(set([y["key"] for y in x.supporting_facts]))).with_inputs("claim")
        )
hover = filtered_hover

trainset, devset, testset = hover[:100], hover[100:150], hover[650:]

example = trainset[0]

print("Claim:", example.claim)
print("Pages that must be retrieved:", example.titles)

DOCS = {}


class SearchInput(BaseModel):
    query: str


def search(query: str, k: int) -> list[str]:
    results = dspy.ColBERTv2(url="http://20.102.90.50:2017/wiki17_abstracts")(query, k=k)
    results = [x["text"] for x in results]

    for result in results:
        title, text = result.split(" | ", 1)
        DOCS[title] = text

    return results


def search_wikipedia(query: SearchInput) -> list[str]:
    """Returns top-5 results and then the titles of the top-5 to top-30 results."""

    topK = search(query.query, 30)
    titles, topK = [f"`{x.split(' | ')[0]}`" for x in topK[5:30]], topK[:5]
    return topK + [f"Other retrieved pages have titles: {', '.join(titles)}."]


def lookup_wikipedia(title: str) -> str:
    """Returns the text of the Wikipedia page, if it exists."""

    if title in DOCS:
        return DOCS[title]

    results = [x for x in search(title, 10) if x.startswith(title + " | ")]
    if not results:
        return f"No Wikipedia page found for title: {title}"
    return results[0]


instructions = "Find all Wikipedia titles relevant to verifying (or refuting) the claim."
signature = dspy.Signature("claim -> titles: list[str]", instructions)

tools = [dspy.Tool(search_wikipedia), dspy.Tool(lookup_wikipedia)]

codeact = dspy.CodeAct(signature, tools=tools, max_iters=20)

output = codeact(claim="David Gregory was born in 1625.")

Error:

dspy.primitives.python_interpreter.InterpreterError: NameError: ["name 'SearchInput' is not defined"]

Looks like we also need to register the custom types to the interpreter.

I tried to change query: SearchInput to query: str, but doesn't work either with the following error:

[[ ## observation_18 ## ]]
Failed to execute the generated code: NameError: ["name 'search' is not defined"]

search is a helper function called inside search_wikipedia(query: str), and the registration misses it.

[TomeHirata]

@chenmoneygithub Thanks for trying this out. That is a known limitation where Tool functions can only depend on built-in naming. In other words, tools should be self-contained. We could re-register non-tool variables in locals(), but I worry that this causes unnecessary latency or failures of the re-registration due to external packages or definition ordering. Since LLM can generate code rather than json to pass args to the tools, I don't think this is the blocking limitation. The langgraph-codeact also has this limitation.