swelancer

SWE-Lancer

This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?".

Note: as of 2025/07/17, this repo contains a subset of the original 237 problems mentioned in the SWE-Lancer paper. Specifically, this repo has 198 tasks that were adjusted and verified to run successfully offline. We have dropped the remaining 39 problems.

Setup

Package Management and Requirements

Install SWELancer with uv

UV_GIT_LFS=1 uv sync

Docker Setup

Tasks in SWELancer rely on Docker images. You can either run SWELancer with a single "monolith" image, where task-specific setup will occur at run-time, or you can build task-specific images for each task.

To build task-specific images (e.g., for issue 28565_1001):

uv run python scripts/build_images.py 28565_1001 --skip-push

Or omit the issue entirely to build images for all tasks:

uv run python scripts/build_images.py --skip-push

Note that while there is some concurrency here, you may want to write your own script for bulk-building and pushing per-task images, depending on your systems. Each task image takes 10-20 minutes to build, and occupies ~14GB

To build the "monolith" image run

uv run python scripts/build_images.py monolith --skip-push

To use it, make sure to set swelancer.use_single_image=True in your run command, below.

Note, if you do not want to skip pushing the images, then you should remove the --skip-push argument and provide a --registry argument.

Note, we have also already pushed the images to dockerhub, which can be found here.

Note, to run SWE Manager tasks, you must use the "monolith" image.

Environment variables

There are a couple of environment variables SWELancer relies on.

To set them, locate the sample.env file in the root directory. This file contains template environment variables needed for the application:

# sample.env contents example:
OPENAI_API_KEY=
OPENROUTER_API_KEY=

USE_WEB_PROXY=false
EXPENSIFY_URL=https://www.expensify.com/
NEW_EXPENSIFY_URL=https://new.expensify.com/

Create a new file named .env and copy the contents from sample.env, setting the appropriate values for each variable. This file will be used to configure the application. Usually you just need to set OPENAI_API_KEY or OPENROUTER_API_KEY to use the OpenAI or OpenRouter API, respectively.

Running SWELancer

Note: SWELancer is designed to run with internet disabled. At the moment, this is only supported on Linux systems, as it relies on iptables. It is possible to run on MacOS by adding the swelancer.disable_internet=False argument to any run commands. However, note that we have observed that some tasks behave abnormally when run with internet enabled, and we only consider rollouts run with internet disabled to be valid. Finally, while disabling internet should not have undesired side effects on the host machine, because of the nature of running iptables commands, we recommend running swelancer on an ephemeral VM. If you notice any weird effects, try running sudo iptables -S, and delete any rules that have the comment alcatraz_block.

We've implemented a DummySolver which can be used to verify that the evaluation works as intended:

• Enabling the test_user_tool argument will make the dummy solver use and log the user tool. This can take a while to execute (5-10 mins)
• Enabling the apply_gold_solution argument will make the dummy solver apply the "gold" solution to the task. By default the dummy solver doesn't modify the codebase, so you should expect it to fail unless you've enabled this argument

Below is an example of how to run the dummy solver on IC SWE Diamond, when not having it test the user tool, but making it apply the gold solution:

uv run python swelancer/run_swelancer.py \
  swelancer.split=diamond \
  swelancer.task_type=ic_swe \
  swelancer.solver=swelancer.solvers.dummy.solver:DummySolver \
  swelancer.solver.test_user_tool=False \
  swelancer.solver.apply_gold_solution=True \
  swelancer.solver.computer_runtime=nanoeval_alcatraz.alcatraz_computer_interface:AlcatrazComputerRuntime \
  swelancer.solver.computer_runtime.env=alcatraz.clusters.local:LocalConfig \
  swelancer.solver.computer_runtime.env.pull_from_registry=True \
  swelancer.docker_image_prefix=swelancer/swelancer_x86 \
  swelancer.docker_image_tag=releasev1 \
  runner.concurrency=20 \
  runner.experimental_use_multiprocessing=False \
  runner.enable_slackbot=False \
  runner.recorder=nanoeval.recorder:dummy_recorder \
  runner.max_retries=2

Same as above but for a single issue (e.g., 28565_1001):

uv run python swelancer/run_swelancer.py \
  swelancer.split=diamond \
  swelancer.task_type=ic_swe \
  swelancer.taskset="['28565_1001']" \
  swelancer.solver=swelancer.solvers.dummy.solver:DummySolver \
  swelancer.solver.test_user_tool=False \
  swelancer.solver.apply_gold_solution=True \
  swelancer.solver.computer_runtime=nanoeval_alcatraz.alcatraz_computer_interface:AlcatrazComputerRuntime \
  swelancer.solver.computer_runtime.env=alcatraz.clusters.local:LocalConfig \
  swelancer.solver.computer_runtime.env.pull_from_registry=True \
  swelancer.docker_image_prefix=swelancer/swelancer_x86 \
  swelancer.docker_image_tag=releasev1 \
  runner.concurrency=20 \
  runner.experimental_use_multiprocessing=False \
  runner.enable_slackbot=False \
  runner.recorder=nanoeval.recorder:dummy_recorder \
  runner.max_retries=2

We've implemented a SimpleAgentSolver which interacts with the codebase over multiple steps to solve the task:

• Change the model argument to use different models. OAI and OpenRouter models are supported. The model argument should follow the format <PROVIDER>/<MODEL>, for example, openai/gpt-4o, or openrouter/anthropic/claude-3.5-sonnet

Below is an example of how to run the SimpleAgentSolver on IC SWE Diamond with gpt-4o:

uv run python swelancer/run_swelancer.py \
  swelancer.split=diamond \
  swelancer.task_type=ic_swe \
  swelancer.taskset="['28565_1001']" \
  swelancer.solver=swelancer.solvers.swelancer_agent.solver:SimpleAgentSolver \
  swelancer.solver.model=openai/gpt-4o \
  swelancer.solver.computer_runtime=nanoeval_alcatraz.alcatraz_computer_interface:AlcatrazComputerRuntime \
  swelancer.solver.computer_runtime.env=alcatraz.clusters.local:LocalConfig \
  swelancer.solver.computer_runtime.env.pull_from_registry=True \
  swelancer.docker_image_prefix=swelancer/swelancer_x86 \
  swelancer.docker_image_tag=releasev1 \
  runner.concurrency=4 \
  runner.experimental_use_multiprocessing=False \
  runner.enable_slackbot=False \
  runner.recorder=nanoeval.recorder:dummy_recorder \
  runner.max_retries=2

To run manager tasks, set swelancer.task_type=swe_manager. Currently, swe_manager tasks are only supported using a monolith image, so you must also set swelancer.use_single_image=True. Below is an example of how to run SimpleAgentSolver on SWE Manager Diamond with gpt-4o for a specific task (e.g. 16921-manager-0):

uv run python swelancer/run_swelancer.py \
  swelancer.split=diamond \
  swelancer.task_type=swe_manager \
  swelancer.taskset="['16921-manager-0']" \
  swelancer.solver=swelancer.solvers.swelancer_agent.solver:SimpleAgentSolver \
  swelancer.solver.model=openai/gpt-4o \
  swelancer.solver.computer_runtime=nanoeval_alcatraz.alcatraz_computer_interface:AlcatrazComputerRuntime \
  swelancer.solver.computer_runtime.env=alcatraz.clusters.local:LocalConfig \
  swelancer.solver.computer_runtime.env.pull_from_registry=True \
  swelancer.docker_image_prefix=swelancer/swelancer_x86 \
  swelancer.docker_image_tag=releasev1 \
  swelancer.use_single_image=True \
  runner.concurrency=4 \
  runner.experimental_use_multiprocessing=False \
  runner.enable_slackbot=False \
  runner.recorder=nanoeval.recorder:dummy_recorder \
  runner.max_retries=2

Logging

By default logs appear under the runs directory. Each evaluation is identified as a "run group" and has a unique run group ID; a directory for each run group is made under the runs directory, and a directory for each evaluated task is made inside the run group directory.