Welcome to Taiga! Taiga is a specification and platform for building RL environments. In these environemnts, a model automously works on a task, and the outcome of it's work on this task is graded.
Staying up to date: You can always download the latest version of this repository from taiga.ant.dev/api/taiga-repo/download (requires Taiga authentication). To check if you're on the latest version, see the version endpoint.
taiga-core/- The core TAIGA Python package that provides base classes and utilities for building MCP-based environments.examples/- Example environments demonstrating different problem types (see details below)local-tunnel/- CLI tool for testing environments locally against the hosted Taiga platform (recommended)mcp_client/- Local-only development tool for testing environments interactively with Claude
The local tunnel connects your local Docker container to the hosted Taiga platform at taiga.ant.dev, so you develop and test against the same production environment. See local-tunnel/README.md for full details.
The repository root includes a Makefile for the most common development tasks.
Choose an example with EXAMPLE=<directory-name> and, when needed, a problem
with PROBLEM=<problem-id>:
# Show all commands and list available examples
make help
make examples
# Install taiga-core test dependencies and the local tunnel CLI
make setup
make tunnel-login
# Run unit tests
make test
# Build, validate, and run a specific example
make example-prepare EXAMPLE=dmcc
make example-build EXAMPLE=dmcc
make example-validate EXAMPLE=dmcc
make example-run EXAMPLE=dmcc PROBLEM=python1575aThe default example is math-in-python. Override the local image name with
IMAGE=<tag>, pass extra Docker build flags with BUILD_ARGS='...', and use
make example-info EXAMPLE=<name> to inspect the available problem IDs.
Examples with a repository-provided build.sh use that script automatically.
The tunnel-login and example-run targets automatically export variables
from the root .env file; override its location with ENV_FILE=<path>.
The setup targets install into taiga-core/.venv without creating a
repository uv.lock.
Run from the root of this repository:
uv pip install -e local-tunnel/
taiga-local-tunnel login(If you don't use uv, pip install -e local-tunnel/ works too.)
cd examples/math-in-python
docker build --build-context taiga=../../ --platform linux/amd64 -t taiga-math-test -f Dockerfile .
docker run -d -i --name math-test taiga-math-test sleep infinity
taiga-local-tunnel start --container math-test --startup-command "math_in_python mcp" --metadata-file problems-metadata.json --problem-id factorialThe browser opens automatically. Claude will work on the factorial problem from the math-in-python example.
Mount your source code so changes take effect without rebuilding:
docker run -d -i --name my-env \
-v $PWD/src:/mcp_server/environments/my-env/src \
my-env sleep infinityEdit code locally, then start a new session in the browser. Only rebuild when changing dependencies or system packages.
To drive the loop programmatically (trigger model sampling, poll for completion, read grade/transcript) instead of clicking in the browser, see Programmatic control.
docker stop math-test && docker rm math-testThe MCP client is a local-only tool that runs Claude directly against your container. It does not use the hosted Taiga platform.
The best way to get started is to explore an existing example environment:
- Look at an example from the ./examples directory that matches your needs
- Follow the README in that example to build and run the Docker container
- Test the example environment using the mcp_client tool to see Claude work on tasks
After that, start building your own environment!
The examples are representative of different types of envrionments, and include environments that accomplish tasks with Computer Use, Programming/Coding, and also environments that use LLM-As-A-Judge for grading an outcome.
- ./examples/math-in-python - An environment for solving math problems with Python. This is the simplest and most instructive example; it also shows how to use additional features such as task parametrization in the metadata file with
extra_fields, and better debugging of on-container files withoutput_directory. - ./examples/process-monitoring - Long-running and interactive tasks that require a persistent terminal. Demonstrates the
tmuxtool and how to build/test tmux-dependent environments locally. - ./examples/dmcc - An environment for solving DeepMind CodeContests problems in Python and C++
- ./examples/galculator-cu-python - A Computer Use environment that lets Claude use Galculator, an open source scientific caluclator, to solve math problems. This example also shows how to include images in your prompt.
- ./examples/wordpress-cu - A Computer Use environment that lets Claude work on a WordPress website, completing tasks like publishing posts
- ./examples/rubric - An LLM-as-a-Judge environment that asks Claude to work with Word Documents, PowerPoints, etc., and grades the outcomes of tasks with a rubric and LLM-As-A-Judge
Make sure Docker is installed.
math-in-python is a simple environment we can use to watch Claude solve math problems with Pyton.
To build, run:
cd examples/math-in-python
# This command may take a while!
docker build --build-context taiga=../../ -t math-in-python -f Dockerfile .mcp_client is a tool you can use to watch Claude work on a problem.
Start it now by running
cd mcp_client
# Install all dependencies (backend + frontend)
make setup
# Build the frontend for production
make build-frontend
# Start the combined server
make runFinally, open a new terminal and run the following to watch Claude work on your problem:
docker_id=$(docker run -d -i math-in-python uv --offline --directory /mcp_server run math_in_python mcp) &&
open -a "Google Chrome" "http://localhost:5000/app?container_id=$docker_id&problem_id=factorial&max_tokens=64000"Claude will work on the factorial problem from the math-in-python example. You can send messages to Claude during problem solving to interject with guidance or corrections.
An environment is a combination of:
- a Docker container exposing tools to the model as a stdio MCP server
- a set of problems that the container can set up and grade
The MCP server must implement at least the setup_problem and grade_problem tools. The flow for running a problem is:
- Taiga calls
setup_problemwith a problem id, plus optional metadata - The container sets up the environment for the problem, and returns a problem statement prompt
- The model works on the problem, calling tools as needed
- When the model is finished (or runs out of tokens), Taiga calls
grade_problemwith the transcript of the model's work
For example for a computer use environment teaching Claude to make powerpoints:
setup_problem(problem_id: "1")might open PowerPoint and create a new blank presentation, and return a prompt like "Create a presentation with at least 10 slides"- The container might also expose tools like
powerpoint_add_slide,powerpoint_edit_slideetc. which the model uses to make the presentation - When the model is finished, Taiga calls
grade_problem, and the container grades the presentation the model made e.g. by running some logic to check the number of slides, etc.
Once you've explored an example and understand the environment setup, you can build your own environment.
The steps to build an environment are:
- Copy an existing example that's similar to the environment you want to build
- Create new problems for the model to work on, and modify the
setup_problemandgrade_problemfunctions - Update the Docker configuration as needed
- Test using the mcp_client debugging workflow
The container image holds machinery (tools, grading logic, system packages). The per-problem data (the prompt text, the expected answer, the rubric) lives in problems-metadata.json under each problem's extra_fields key, and Taiga passes it to your MCP server at runtime as the extra_fields argument to setup_problem and grade_problem.
This means you can add or edit problems without rebuilding and re-pushing the Docker image — you just edit the metadata in Taiga.
@mcp.tool()
async def setup_problem(problem_id: str, extra_fields: dict | None = None, ...) -> str:
# extra_fields comes from problems-metadata.json -> problems[i].extra_fields
return extra_fields["statement"]
@mcp.tool()
async def grade_problem(problem_id: str, transcript: str, extra_fields: dict | None = None) -> Grade:
expected = extra_fields["solution"]
return Grade(subscores={"match": float(expected in transcript)}, weights={"match": 1})For things that genuinely can't be data — e.g. a grading function that inspects live container state — keep the function in the image and reference it by name from extra_fields (see the GRADERS registry in wordpress-cu).
extra_fields is a JSON column, so it suits small values (prompts, expected answers, config). When per-problem data is large — e.g. a competitive-programming test suite or a fixture spreadsheet — attach it as a file instead. Each problem can list preloaded_files, which Taiga uploads and copies into the container at startup; extra_fields then carries just the in-container path.
{
"id": "python1575a",
"extra_fields": {"language": "python", "data_file": "/tmp/files/problem.json"},
"preloaded_files": [
{"local_path": "data/problems/1575a.json", "remote_path": "/tmp/files/problem.json"}
]
}@mcp.tool()
async def setup_problem(problem_id: str, extra_fields: dict | None = None) -> str:
with open(extra_fields["data_file"]) as f:
record = json.load(f)
...See dmcc for a worked example.
Let's use math-in-python as an example.
First, copy math-in-python like:
mkdir -p ./environments
cp -r ./examples/math-in-python/ ./environments/math-in-python-v2/Next, add a new problem. Problem data lives in problems-metadata.json under extra_fields — the container reads it at runtime, so you don't need to touch Python to add a problem.
Open ./environments/math-in-python-v2/problems-metadata.json and add an entry to the problems array:
{
"image": "math-in-python-v2",
"startup_command": "uv --offline --directory /mcp_server run math_in_python mcp",
"id": "exponential",
"required_tools": ["bash", "str_replace_editor"],
"extra_fields": {
"statement": "compute 10^2",
"solution": "100"
}
}setup_problem will read extra_fields["statement"] to build the prompt, and grade_problem will read extra_fields["solution"] to check the answer. You only edit Python when you want to change how prompts are built or how answers are graded — not to add another problem.
For example, to change grading from substring-match to "run the file the model wrote", edit grade_problem in ./environments/math-in-python-v2/src/math_in_python/math_in_python.py:
import re
results = re.findall(r"<answer>(.*?)</answer>", transcript)
answer = exec(open(f"{results[-1]}").read()) # very naive — for illustration only
expected = _get_problem(problem_id, extra_fields).solution
score = float(str(answer) == expected)
return Grade(subscores={"matched_solution": score}, weights={"matched_solution": 1})Now, to test!
First we package the environment by running
cd ./environments/math-in-python-v2
docker build --build-context taiga=../../ --platform linux/amd64 -t math-in-python-v2 -f Dockerfile .Next we start the mcp_client by running:
cd mcp_client
# Install all dependencies (backend + frontend)
make setup
# Build the frontend for production
make build-frontend
# Start the combined server
make runAnd finally open a new terminal and run
docker_id=$(docker run -d -i math-in-python-v2 uv --offline --directory /mcp_server run math_in_python mcp) &&
open -a "Google Chrome" "http://localhost:5000/app?container_id=$docker_id&problem_id=exponential&max_tokens=64000"An important note, this part of the command
uv --offline --directory /mcp_server run math_in_python mcpis starting the MCP server defined in the math_in_python example. Claude will communicate with this server like any other MCP server, calling tools and responding to tool calls.
If we are modifying the grade_problem, setup_problem, or other code, we can also mount our code instead of having to build the Docker container after every single change, like so:
docker_id=$(docker run -d -v ./environments/math-in-python-v2/src:/mcp_server/environments/math-in-python-v2 -i math-in-python-v2 uv --offline --directory /mcp_server run math_in_python mcp) &&
open -a "Google Chrome" "http://localhost:5000/app?container_id=$docker_id&problem_id=exponential&max_tokens=64000"After iterating on your envrionment, before submitting, you can run the following validation tests:
cd taiga-core
uv venv
source .venv/bin/activate
uv pip install ".[dev]"
# Run validation tests for an environment
pytest -m validate_env --problems-metadata-path=/path/to/problems-metadata.jsonFirst, you need to build the docker container to target linux/amd64.
So rebuild your docker environment with the following command, which uses the flag --platform linux/amd6
docker build --build-context taiga=../../ -t <replace-me-with-actual-name-of-env> --platform linux/amd64 -f Dockerfile .Important
Make sure to replace with the actual name your environment
You will need push your finished Docker container to the Taiga Docker registry. Then, you will need to create a problems-metadata.json and submit it to https://taiga.ant.dev/.
More details in the sections below.
The specification should contain a list of all problem ids and associated metadata in the following JSON schema format:
{
"problem_set": {
"properties": {
"owner": {
"type": "string",
"description": "The owner/creator of this problem set (e.g., 'feingold')"
},
"name": {
"type": "string",
"description": "Name of the problem set (e.g., 'math-problems')",
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
},
"description": {
"type": "string",
"description": "Detailed description of what this problem set contains and tests (e.g., 'Basic math problems including factorial and GCD calculations')"
},
"required_resources": {
"type": "string",
"enum": [
"2vcpu+6gib",
"4vcpu+16gib",
"6vcpu+32gib",
"8vcpu+64gib",
"16vcpu+64gib",
"13vcpu+32gib+tpuv5e1x1",
"50vcpu+128gib+tpuv5e2x2",
"3vcpu+25gib+h100/8",
"6vcpu+50gib+h100/4",
"12vcpu+100gib+h100/2",
"24vcpu+200gib+h100/1"
],
"default": "2vcpu+6gib",
"description": "Optional amount of memory and vCPUs to run the problems with. GPU tiers (h100/*) give access to H100 GPUs with different oversubscription ratios. Defaults to 2vcpu+6gib."
},
"version": {
"type": "string",
"description": "Version of this problem set for tracking updates (e.g., '1.0.0')"
},
"created_at": {
"type": "string",
"format": "date-time",
"description": "ISO 8601 timestamp when this problem set was created"
},
"metadata": {
"type": "object",
"description": "Optional metadata for the entire problem set (e.g., categories, tags, difficulty)",
"additionalProperties": true
},
"problems": {
"type": "array",
"items": {
"type": "object",
"properties": {
"image": {
"type": "string",
"description": "Docker image containing the environment for this problem (e.g., '377799100787.dkr.ecr.us-east-1.amazonaws.com/sandboxing-container-server@sha256:a1b2c3d4...')"
},
"startup_command": {
"type": "string",
"description": "Command to start the MCP server in the container (e.g., 'python -u /mcp_server/math_in_python.py')"
},
"id": {
"type": "string",
"description": "Unique identifier for this specific problem (e.g., 'factorial', 'gcd')",
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
},
"extra_fields": {
"type": "object",
"description": "Additional configuration fields to be interpreted by setup and grading functions on the MCP server. These fields will be passed directly to the MCP server on both setup and grading calls.",
"additionalProperties": true
},
"metadata": {
"type": "object",
"description": "Arbitrary labeling/metadata fields like difficulty level, task type, or task size",
"additionalProperties": true
},
"required_tools": {
"type": "array",
"items": {
"type": "string",
"enum": ["bash", "str_replace_editor", "computer", "python_exec", "memdir", "browser", "javascript_exec", "tmux", "message_search_raive", "web_fetch_research", "web_fetch_search_unified", "web_search_research", "web_search_deep_research", "web_search_deep_research_asi"]
},
"description": "[ONLY AFFECTS HOSTED TOOLING] List of which of our pre-baked tools should be used for this problem. Common tools: 'bash' (shell commands), 'str_replace_editor' (file editing), 'computer' (GUI interaction), 'browser' (web browsing), 'python_exec' (Python execution). On local tooling, use tools from taiga-core/src/taiga/tools/ instead."
},
"enable_anthropic_api": {
"type": "boolean",
"default": false,
"description": "[ONLY AFFECTS HOSTED TOOLING] Enable usage of the Anthropic API for this problem, for things like grader models."
},
"enabled_package_managers": {
"type": "array",
"items": {
"type": "string",
"enum": ["npm", "cargo", "pip", "maven", "conda", "yarn", "apt"]
},
"default": [],
"description": "[ONLY AFFECTS HOSTED TOOLING] List of package managers enabled for this problem. Default is empty list."
},
"tool_timeout_seconds": {
"type": "integer",
"minimum": 1,
"maximum": 3600,
"default": 300,
"description": "[ONLY AFFECTS HOSTED TOOLING] Timeout in seconds for tool execution in this problem. Defaults to 300 seconds (5 minutes). Maximum is 3600 seconds (1 hour)."
},
"setup_timeout_seconds": {
"type": "integer",
"minimum": 1,
"maximum": 1800,
"default": 300,
"description": "[ONLY AFFECTS HOSTED TOOLING] Timeout in seconds for awaiting a response to setup_problem in this problem. Defaults to 300 seconds (5 minutes). Maximum is 1800 seconds (30 minutes)."
},
"grading_timeout_seconds": {
"type": "integer",
"minimum": 1,
"maximum": 5400,
"default": 600,
"description": "[ONLY AFFECTS HOSTED TOOLING] Timeout in seconds for awaiting a response to grade_problem in this problem. Defaults to 600 seconds (10 minutes). Maximum is 5400 seconds (1.5 hours)."
},
"domain_allowlist": {
"type": "array",
"items": {
"type": "string"
},
"default": [],
"description": "[ONLY AFFECTS HOSTED TOOLING] List of domains that the problem environment is allowed to access. Use this to restrict network access to specific APIs or services (e.g., ['api.github.com', 'pypi.org', '*.googleapis.com']). To allowlist a domain, Anthropic must also approve and add it to our global allowlist. Please contact us."
},
"output_directory": {
"type": "string",
"description": "[ONLY AFFECTS HOSTED TOOLING] Filepath on the container that you would like to extract at the end of the rollout. Useful for debugging in hosted tooling."
},
"container_runtime": {
"type": "string",
"enum": ["gvisor", "firecracker"],
"default": "gvisor",
"description": "[ONLY AFFECTS HOSTED TOOLING] Container runtime to use. gVisor (default) provides strong isolation. Firecracker uses microVMs. TPU/GPU workloads require gVisor."
},
"grading_strategy": {
"type": "object",
"description": "[ONLY AFFECTS HOSTED TOOLING] Grading strategy configuration. Types: 'mcp' (MCP server grading), 'rubric' (LLM-as-judge rubric), 'junit' (JUnit test results), 'office_rubric' (Office document rubric), 'basic' (bash command grading), 'rubric_advisor' (rubric with advisor).",
"properties": {
"type": {
"type": "string",
"enum": ["mcp", "rubric", "junit", "office_rubric", "basic", "rubric_advisor"]
},
"weight": {
"type": "number",
"default": 1.0
}
}
}
},
"required": ["image", "startup_command", "id", "required_tools"]
}
}
},
"required": ["owner", "name", "description", "problems"]
}
}An example metadata schema for math_in_python.py can be found in examples/math-in-python/problems-metadata.json.
The taiga environment must be provided as a docker image, which must use an amd64 based image (no arm64 images).
You must pick one of the following sizes: 2vcpu+6gib, 4vcpu+16gib, 6vcpu+32gib.
The image must include all data and dependencies for environment and problems - no internet access.
There is an Artifactory repository that proxies all dependency requests on the host artifactory.local.
You may not run docker in this environment.
The docker container should run a MCP Server.
Your docker image should have the following users:
- a root user, which should have read/write/execute privileges for the MCP server, problem setup files, etc
- a non-privileged user, which the model will use, which should not have read/write/execute privileges for the MCP server, problem setup files, etc. This is to ensure that the model cannot "hack" the environment by accessing grading logic or changing problem files. This user should have uid and gid 1000
If for any reason you need to use other users outside of the above two, ensure that those users are created after the root user and the model's non-privilged user. This is to ensure the model gets uid and gid 1000 for its user.
The MCP Server should implement the following tools:
Configures the environment for a specific problem. This will only be called once on a fresh container. Enviroments that need to run services in the background should do so here. This tool must return a string containing the problem statement - the prompt provided to the model describing the task. In addition to the task, this problem statement should contain any additional instructions the model should follow. For example, if the problem requires the model to return its answer surrounded by tags, that should be specified as part of the problem statement.
Grades the solution to a specific problem. The input to this tool is the entire transcript produced by the model and its tool calls
as part of the interaction. The input may be discarded if grading relies on filesystem or database state.
Note that the MCP server might be restarted between calls to setup_problem and grade_problem to free up memory.
This tool should either return:
- a float representing the grade, which must be between [0,1] inclusive
- a dictionary of the shape:
where:
{ "subscores": { "subscore1": 1, "subscore2": 0, "subscore3": 0.5, }, "weights": { "subscore1": 0.5, "subscore2": 0.2, "subscore3": 0.3, }, }- the keys in
subscoresandweightsmust be the same - each subscore must be between [0,1] inclusive
- the sum of all weights must be 1
- the keys in
Set enable_anthropic_api: true on an individual problem to enable usage of the Anthropic API when grading a problem.
You do not need to worry about setting an API key (except when testing locally).
When run on our taiga website, the API key will automatically be set for you.
Use this sparingly, and always reach for a deterministic, non-LLM grading solution first.
setup_problem exposes a use_hinted_problem field, default true, which toggles between a hinted version of the problem and an un-hinted version.
The hinted version, if you choose to provide one, should give the model some additional clues in the prompt
to help it solve the problem.
Important: please only use this functionality to add hints to the prompt. All other problem setup should be exactly the same.
By default, we will run your problems in hinted mode (use_hinted_problem == true) in our tooling,
and all problem verification and pass@ distributions will be done from the hinted version.
These tools are included in the reference implementation and can be used to test the environment locally with an MCP host such as Claude Desktop. These tools will be replaced with our own implementation in training.
If your environment requires the tmux tool (long-running or interactive processes), your Docker image should
install tmux.
If you're environment requires the computer tool, your Docker image should install
xdotool, scrot, and imagemagick to ensure it will work correctly.
You may find it easier to iterate on your problems in a mode where your problems are defined somewhere in your code rather than in a separate metadata file. For this, you can provide a 'list_problems' tool in your MCP server that returns a jsonschema with an enum of available problem id's, such as:
{
"properties": {
"id": {
"enum": ["problem-id-1", "problem-id-2"]
}
}
}You are allowed to specify additional tools for the model to use as part of your MCP server. Below are some restrictions and best practices for specifying these tools. The example MCP server in this repo has some examples of valid custom tools.
- Only a single MCP server is supported; all custom tools must be attached to the primary MCP server
- Tool params must only have a single type
- Valid params
value: strvalue: str | None
- Invalid params
value: str | list[str]value: str | float | None
- Valid params
- Tool responses must include
isError=truewhen a tool call errors, as described in the MCP tools spec - Tools cannot interact with external resources
- Tools can only return string values - returning images is not supported
- Your tools'
inputSchemas should be well-documented- This include well documented tool definitions, param definitions, and param input types
- When params are not globally required, but are required in conjunction with other params, you should specify as such in the param's description
- Call
list_toolson your server and review theinputSchemas for your custom tools to ensure they make sense, correctly indicate required parameters, etc - Robustly validate tool inputs when your tool is called, and raise errors when inputs are not valid
- Your custom tools should be restrictive in capabilities and not give the model broad access to run arbitrary commands on the container. Since these tools run on the same server as problem setup and grading, tools with broad capabilities could inadvertently give Claude root access to the container
The mcp server should be started as root and all files and processes that define the problems, solutions and execution environment should be owned by root. The problem will be solved by a non root user (uid=1000, gid=1000) to prevent it from accessing problem data. You must take special care to:
- Disallow the model from reading or writing to problem data
- Check that user 1000:1000 can solve the problem
The reference implementation demonstrates this by dropping down to 1000:1000 in all tool calls.
For Computer Use tasks, we have added an Addendum here with a word on setup as well as task selection.
Taiga uses gVisor's runsc runtime by default for enhanced container security isolation. Set CONTAINER_RUNTIME=runc to use the standard Docker runtime instead. gVisor only runs on Linux — macOS users must use CONTAINER_RUNTIME=runc.
Install on Ubuntu/Debian:
curl -fsSL https://gvisor.dev/archive.key | sudo gpg --dearmor -o /usr/share/keyrings/gvisor-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases release main" | sudo tee /etc/apt/sources.list.d/gvisor.list > /dev/null
sudo apt-get update && sudo apt-get install -y runsc
sudo runsc install
sudo systemctl reload docker
docker run --rm --runtime=runsc hello-world- Builds are slow (~30 seconds due to package installation) — batch changes before rebuilding.
- Rebuild after changing problem definitions, grading logic, or entrypoint scripts. Hot-reload (volume-mount your
src/) avoids most rebuilds. - New environments go in
environments/, notexamples/. Theexamples/directory is for stable reference implementations.
Each environment is built from within its own directory using multiple build contexts (Docker CLI 20.10.13+, BuildKit, # syntax=docker/dockerfile:1.4):
# Copy taiga core package from named 'taiga' context (../../)
COPY --from=taiga taiga-core ./taiga-core
# Copy this environment from default context (current directory)
COPY . ./environments/<env-name>
RUN uv pip install --system -e ./taiga-core
RUN uv pip install --system -e ./environments/<env-name>Never use FROM taiga — it doesn't install packages system-wide. Always copy from an existing example (examples/dmcc/Dockerfile, examples/rubric/Dockerfile, examples/galculator-cu-python/Dockerfile) and follow the pattern: FROM docker.io/ubuntu:22.04 → create uid/gid 1000 user → install Python 3.11 + system packages → multi-context copy + install.
- Problem IDs must match
^[a-z0-9]+(-[a-z0-9]+)*$(no underscores). - Docker build must be run from within the environment directory:
docker build --build-context taiga=../../ -f Dockerfile . - Missing dependencies — check imports match
pyproject.toml(must includetaiga-core,typer,mcp[cli]>=1.3.0,pydantic). - Integration test timeouts — check
docker logs; "Failed to spawn" usually means a package is not installed. - Editable installs — use
-e .for both taiga-core and your package. - TEST_MODE — expose
bash/str_replace_editor/tmuxtools whenMCP_TESTING_MODE=1(see examples).
Transcripts are saved to .transcripts/ in the repository root.
- Run the client (
cd mcp_client && make setup && make build-frontend && ANTHROPIC_API_KEY=... make run). - Execute problems through the client UI; transcripts are saved as
transcript_YYYYMMDD_HHMMSS.json. - Review: filter to lines containing
"text"to see Claude's actual output (grep '"text"' transcript.json); check the final grade and subscores. - Iterate: look for where Claude gets stuck, whether
setup_problemis clear, whethergrade_problemchecks the right things, where partial credit could apply. - Re-run several times to check for non-deterministic failures.
When reading a transcript, ask: is the objective clear? are intermediate steps observable? does grading reward incremental progress? did failures come from hidden preconditions, ambiguous UI, or timing?
- Setup — one todo per problem, fix the model used, clear
.transcripts/, create.transcripts/evaluated/. - Per problem — run via mcp_client → wait →
grep '"text"'the transcript → check grade → move toevaluated/→ decide whether the prompt or the grader (or both) needs changing → batch changes → rebuild → re-test until consistently >80%. - Batch Docker rebuilds, document changes, prefer partial credit over binary pass/fail, accept reasonable variations, check for valid alternative approaches.
Common refinements: more explicit prompts about expected formats; adding partial credit; accepting "around N" instead of exactly N; splitting multi-step tasks into explicit sub-requirements.
When editing taiga-core/src/taiga/tools/:
cd taiga-core
python3 -m pytest tests/test_browser_tool*.py -v # < 1s, run after every changeWhen adding a new action/parameter, add a unit test to tests/test_browser_tool_new_features.py asserting the action is in BrowserAction and the method exists. If new system packages are needed, update tests/test_cu_tool_deps.py and the example Dockerfiles. Full validation (Docker, ~30s):
pytest tests/test_cu_tool_deps.py -v --problems-metadata-path=../examples/wordpress-bu/problems-metadata.jsonUnit tests check API surface; for runtime behaviour use mcp_client and inspect .transcripts/.
# Build (from within environment or example directory)
docker build --build-context taiga=../../ --platform linux/amd64 -t my-env -f Dockerfile .
# Validation tests (gVisor by default; CONTAINER_RUNTIME=runc to use standard runtime)
pytest taiga-core/tests/test_integration.py --problems-metadata-path=environments/<env>/problems-metadata.json
pytest -m validate_env --problems-metadata-path=environments/<env>/problems-metadata.json
# MCP client
ANTHROPIC_API_KEY=your_key make -C mcp_client run
COMPUTER_WIDTH_PX=1400 COMPUTER_HEIGHT_PX=850 ANTHROPIC_API_KEY=your_key make -C mcp_client run # computer use
# Hot-reload
docker run -d -i --runtime=runsc -v $PWD/src:/mcp_server/src my-env <startup_command>- Google Cloud SDK (gcloud) installed
- Docker installed
- Access to your company's Taiga Docker repository. Please reach out to your Anthropic contact to get access.
Set the environment variables for your deployment, and login to Google Artifact Registry:
# Replace with your repository name
# This might have been given you to as a URL like "us-east1-docker.pkg.dev/gcp-taiga/your-company-name"
# Anthropic employees should use "internal" here
export REPOSITORY_NAME="your-company-name"
# Environment name
export ENV_NAME="my-exciting-env"
# Version tag (defaults to timestamp)
export VERSION="$(date +%Y%m%d-%H%M%S)"
gcloud auth login
gcloud auth configure-docker us-east1-docker.pkg.dev# Navigate to your Dockerfile directory
cd environments/$ENV_NAME
# Build the image
docker build --build-context taiga=../../ --platform linux/amd64 -t $ENV_NAME -f Dockerfile .Tag your image with a specific version number/identifier. We do not support images tagged as latest.
docker tag $ENV_NAME us-east1-docker.pkg.dev/gcp-taiga/$REPOSITORY_NAME/$ENV_NAME:$VERSIONdocker push us-east1-docker.pkg.dev/gcp-taiga/$REPOSITORY_NAME/$ENV_NAME:$VERSIONgcloud artifacts docker images describe us-east1-docker.pkg.dev/gcp-taiga/$REPOSITORY_NAME/$ENV_NAME:$VERSIONTo pull and use the image:
docker pull us-east1-docker.pkg.dev/gcp-taiga/$REPOSITORY_NAME/$ENV_NAME:$VERSION