Runt AI Agent evaluations
With the addition of data directory mounts and vector store embedding the Runt Pyodide Agent has gained capability similar to OpenAI Code Interpreter but all code generated and run by the AI agent remains available in the Anode notebook.
This work uses the DSBench evaluation framework to determine the effectiveness of the Runt Pyodide Agent to create and execute Python code to solve data modeling tasks. The evaluations provided can be directly compared to the published DSBench results for the OpenAI Code Interpreter.
DSBench aims to evaluate not just an LLM but a holistic Data Science Agent that takes multi-modal inputs and instructions and only the final outcome is evaluated against a human-generated ground-truth result. For data modeling tasks all data sets come from Kaggle competitions the ground truth result is the published score of the competition winner. The details of what code was generated or how long it took are not directly evaluated, only the final result. The final score assigned to the DS Agent result for each data set is a relative improvement ratio or relative performance gap (RPG) of the DS Agent result from a baseline score to the Kaggle winner score.
RPG = (ds_agent_score - baseline_score) / (kaggle_winner_score - baseline_score)
This allows for normalization of the individual scores for each competition dataset, which are each of themselves computed with different machine learning metrics (like r-squared, or rmsl, or accuracy). The final score is the average of all relative improvement ratios.
The DSBench eval framework includes 74 data modelling tasks taken from Kaggle competitions. To generate the ds_agent_score for each of the data sets the DS Agent is configured with the following
- The 3 CSV files from the Kaggle dataset
- train.csv, test.csv, sample_submission.csv
- These files may be ingested into a vector store engine
- The full text description of the Kaggle competition taken from the webpage
- This can be provided directly in the AI context
- An activation prompt (All datasets use the same prompt)
You are a data scientist. I have a data modeling task. You must give me the predicted results as a CSV file as detailed in the following content. Don't ask me any questions. I provide you with three files. One is training data, one is test data. There is also a sample file for submission
Then the DS Agent is initiated. If the agent was able to generate an output CSV file with the model predictions it is evaluated against reserved testing data not given to the DS Agent. A separate evaluation script is provided for each dataset with the appropriate ML metric and the RPG is computed. For all successful DS Agent runs the RPG average is computed.
The table below shows the published results from the DSBench authors. For this work we’ll focus on the Code Interpreter GPT-4o. The important columns are Task Success and RPG. Task success refers to the percentage of the 74 datasets for which a prediction CSV file was created and the average RPG for those completed tasks.
To enable automation of DSBench evaluations for Runt, the runt-eval project was used to load cells into a Runt notebook on https://app.runt.run and start execution of the AI cell. The tools used to generate a new notebook, feed the cell inputs, run the notebook, and compute RPG have been added to a fork of the DSBench repository:
https://github.com/AlbertDeFusco/DSBench/tree/runt-eval
To maintain comparison with the published DSBench results we also used GPT-4o Mini
- The Max AI iterations is set to 100
- Cell execution timeout is set to 2000ms
- The timeout of a cell execution triggered through and AI tool is set to 3000000ms
- Only the first 5 rows of the supplied CSV files are indexed into the vector store
- The Runt system prompt with vector store additions and tool descriptions are used as-is
For example here’s a sample of the output from a single dataset using the generate.py runner from the DSBench fork. You’ll see at the bottom that Runt successfully generated the CSV file and the RPG was computed at 0.81. Note that the three data files are mounted into the pyodide runtime and indexed into the vector store engine.
> python generate.py playground-series-s3e20
⏱️ Cell execution timeout: 2000 seconds
⏱️ Runtime timeout: 34 seconds
🔧 Timeout configuration will be passed to runtime agent
📝 Using existing notebook ID...
✅ Notebook 'unknown' created with ID: Ksxgv0IApUtB
🚀 Starting combined automation + runtime
📔 Notebook ID: Ksxgv0IApUtB
🌐 Live URL: https://app.runt.run/nb/Ksxgv0IApUtB
🚀 Starting automation client and runtime agent in parallel...
📔 Using notebook ID: Ksxgv0IApUtB
⏱️ Automation execution timeout: 2000 seconds
⏱️ Automation runtime timeout: 34 seconds
🐍 Starting pyodide runtime agent...
🔍 Environment info:
Deno: 2.4.0
Platform: darwin-aarch64
CI detected: ❌
📁 Ensuring Pyodide cache directory: /Users/adefusco/.runt/pyodide-cache
✅ Cache directory ready
📡 Creating LiveStore connection...
🔗 Sync URL: wss://app.runt.run/livestore
📓 Notebook: Ksxgv0IApUtB
👤 Client: runtime-client
🔐 Auth: Using JWT token
✅ LiveStore created successfully with UserSyncPayload
runtimeAgent sessions: 0
🚀 ✅ Runtime agent connected and ready!
📂 ✅ Vector store ingestion completed successfully
Files indexed: 3
✅ Runtime agent started
runtimeAgent sessions: 1
🔄 Executing document...
🚀 Starting notebook automation via LiveStore
📖 Cells to execute: 2
🔍 Environment debug info:
Deno version: 2.4.0
Platform: darwin-aarch64
Target: aarch64-apple-darwin
Pyodide worker startup environment check...
🏗️ Prefilling notebook structure...
✅ Prefilled notebook structure with 2 cells
🔍 Frontend query result: 2 cells found
- Cell description: fractionalIndex=m, type=markdown
- Cell prompt: fractionalIndex=n, type=ai
⚡ Structure pre-filled - executing cells sequentially with error handling
🔄 Processing cell 1/2: description (markdown)
Source: Welcome to the 2023 edition of Kaggle's Playground Series! Thank you to everyone...
📝 Rendering markdown cell (no execution needed)
✅ markdown cell description rendered successfully
🔄 Processing cell 2/2: prompt (ai)
Source: You are a data scientist. I have a data modeling task.
You must give me the pred...
📤 Submitting execution request for prompt
✅ Execution completed for prompt
✅ Cell prompt completed in 557705ms
📊 Execution summary:
Total duration: 559724ms
Successful cells: 2
Failed cells: 0
📔 Notebook available at: https://app.runt.run/?notebook=Ksxgv0IApUtB
📊 Execution Summary:
Total duration: 557705ms
Successful cells: 2
Failed cells: 0
🌐 View live notebook: https://app.runt.run/?notebook=Ksxgv0IApUtB
🎉 Automation completed successfully!
🧹 Cleaning up...
🧹 Cleaning up automation resources...
📤 Cleaning up 2 subscriptions
🛑 Stopping runtime agent...
✅ Runtime agent stopped
🎯 Process completed successfully
Prediction accuracy against playground-series-s3e20 reserved data: 24.919827772668075
Baseline: 128.28113419454107
Winning Kaggle Score: 0.0
Relative Performance Gap (prediction - baseline) / (kaggle-winner - baseline): 0.8057405094744748
Here are screenshots from the notebook generated for this task. First you’ll see the bottom of the Kaggle competition description added as a markdown cell, which is included in the LLM context. Then the AI cell with the DSBench prompt is executed. Right away you’ll see that the LLM is querying the vector store engine to understand the provided CSV files.
The AI eventually starts creating code cells. Some of them are exploratory cells before making the model. Finally, the model is trained and the submission.csv file is written.
Of the 74 data modeling tasks 61 passed (82%). The average RPG for the completed tasks is 52%.
| Agent | Task Success % | RPG % |
|---|---|---|
| Code Interpreter (GPT 4o mini) | 39.19 | 16.90 |
| Runt Pyodide (GPT 4o mini) | 82.43 | 52.18 |
It is important to point out that everything matters to the DSBench evaluation methodology since the end-to-end result of a DS Agent is evaluated. A DS Agent has many hidden factors that may have outsized influence on the evaluations, perhaps even occluding the influence of the model. Some of these factors may be the following and possibly many more.
- System prompt
- Available tools and their descriptions
- Timeouts on AI tool calls or code execution
- Available memory to the Python interpreter
- OSS packages available to the Python interpreter
More work should be done to identify these common elements of DS Agents and measure their influence on task success and RPG.