Clarifications of the benchmark

[guangyaodou]

Hi,

I am trying to run the benchmark, and have following questions.

1. Where can we find the code_understanding_directory? This seems to be necessary to evaluate data pipeline.
2. What are data sources for each query? For example, for the data_sources of legal-tiny.json, I don't see them being passed as part of the inputs during the generation time. I do see that the only time they were used was format_code_understanding_messages in the gpt_interface file, which was called in generate_key_functionalities_for_workload. However, when and how is the generate_key_functionalities_for_workload being used? I couldn't see any function calls that invoke this method.

Thank you for your time.

[guangyaodou]

Also, shouldn't this example - https://github.com/mitdbg/Kramabench/blob/main/workload/environment.json#L1148-L1163 use string approximate?

[vitaglianog]

Hello @guangyaodou ! Thanks for reaching out and your interest in using KramaBench!
Let me answer the questions:

1. The code_understanding_directory was an artifact required previously for the evaluation of subtasks. However, the content of that folders were equal to the current subtask['step'] in the main workload json files. I pushed a fix to remove the dependency on that folder. Can you let us know if you are able to run the benchmark now?

2. The data sources for each query are part of the ground truth we generated when curating the question set. As such, they should not be seen by the agent generation (or else the benchmark would not test the data discovery capabilities of the solver agents). You are correct in pointing out that we used them in the generate_key_functionalities function, which is a helper function we used in the process of curating the benchmark pipelines to annotate ground truth solutions with subtasks (refer to the paper preprint, Section 3.2).

3. You are correct! Thanks for pointing it out! We are in the process of revising the dataset for such errors, we'll keep you posted when we release a newer version.

[guangyaodou]

Thank you for clarifying! I was able to run kramabench.

Another question I have is that current implementation does not operate within a sandbox environment and generates answers in 'one-shot'. I ended up running models in a sandbox environment and having it explore and reflect on itself; the result is much better than the result showing in the paper (I also relaxed some of the metrics). Any reason why you don't want to put it in a sandbox environment?