Passing variables to the task function in run_experiment

[i59931596-droid]

Describe your question

❓ How to pass custom kwargs to task function in run_experiment?

I'm trying to run an experiment using run_experiment, where the task function needs additional parameters besides the dataset item. Example:

async def classify_label(*, item, model_name, top_k):
    ...

But when running:

langfuse.run_experiment(
    name="Classifier Test",
    data=my_samples,
    task=classify_label,
    model_name="gpt-4",
    top_k=3,
)

The arguments (model_name, top_k) never reach the task function.

SDK Version:
langfuse==3.10.1
(Python package details shown below)

Name: langfuse
Version: 3.10.1
Summary: A client library for accessing langfuse
Author: langfuse
Author-email: [email protected]
License: MIT
Location: /lib/python3.10/site-packages

---

🔍 Issue

Internally, task execution appears to call only:

output = await _run_task(task, item)

Which forwards only item:

async def _run_task(task, item):
    return await task(item=item)

➡ Any kwargs passed to run_experiment() are never forwarded, so the task function cannot use them.

---

📌 Expected Behavior

Since extra parameters can be passed to run_experiment(), I expected that they would be forwarded automatically, e.g.:

output = await _run_task(task, item, **kwargs)

Then the task would work like this:

async def classify_label(*, item, model_name, top_k):
    print(model_name, top_k)  # should be accessible

---

📍 Question

What is the correct way to pass additional arguments to the task function?

• Is there an established pattern to access parameters?
• Or should run_experiment() propagate kwargs in future versions?

Thanks!

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If self-hosted, what version are you running?

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available documentation, the task function in run_experiment receives specific parameters: item, metadata, input, output, and expected_output(1). The documentation shows task functions defined like this:

def my_task(*, item, **kwargs):
    question = item["input"]
    response = OpenAI().chat.completions.create(
        model="gpt-4.1", messages=[{"role": "user", "content": question}]
    )
    return response.choices[0].message.content

(1)

And for evaluators:

def accuracy_evaluator(*, input, output, expected_output, metadata, **kwargs):
    if expected_output and expected_output.lower() in output.lower():
        return Evaluation(name="accuracy", value=1.0, comment="Correct answer found")
    return Evaluation(name="accuracy", value=0.0, comment="Incorrect answer")

(1)

The documentation doesn't show a mechanism for passing custom parameters (like model_name or top_k) directly through run_experiment() to the task function.

Workaround approaches:

1. Use closures - Define your task function inside a wrapper that captures the variables:

def make_task(model_name, top_k):
    async def classify_label(*, item, **kwargs):
        # model_name and top_k are accessible here
        ...
    return classify_label

langfuse.run_experiment(
    name="Classifier Test",
    data=my_samples,
    task=make_task("gpt-4", 3),
)

2. Use metadata - Pass parameters through the metadata configuration option shown in the documentation(1), though this attaches metadata to traces rather than directly passing it to your task function.

The current SDK design appears to only forward the standard parameters to task functions, not arbitrary kwargs passed to run_experiment().

📚 Sources:

1. Experiments via SDK

---

Have another question? Just tag @inkeep.

[Lotte-Verheyden]

Hi @i59931596-droid, the Langfuse SDK does not support additional kwargs for task functions. One workaround is to use closures or partials. You can find an example here.