Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,20 @@
---

# BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

BALROG is a novel benchmark evaluating agentic LLM and VLM capabilities on long-horizon interactive tasks using reinforcement learning environments. Check out how current models fare on our [leaderboard](https://balrogai.com). You can read more about BALROG in our [paper](https://arxiv.org/abs/2411.13543).

## Features

- Comprehensive evaluation of agentic abilities
- Support for both language and vision-language models
- Integration with popular AI APIs and local deployment
- Easy integration for custom agents, new environments and new models

## Installation

We advise using conda for the installation

```bash
conda create -n balrog python=3.10 -y
conda activate balrog
Expand All @@ -26,12 +30,15 @@ cd BALROG
pip install -e .
balrog-post-install
```

On Mac make sure you have `wget` installed for the `balrog-post-install`

## Docker

We have provided some docker images. Please see the [relevant README](docker/README.md).

## ⚡️ Evaluate using vLLM locally

We support running LLMs/VLMs locally using [vLLM](https://github.com/vllm-project/vllm). You can spin up a vLLM client and evaluate your agent on BALROG in the following way:

```bash
Expand All @@ -49,43 +56,53 @@ python eval.py \
```

On Mac you might have to first export the following to suppress some fork() errors:

```
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
```

Check out [vLLM](https://github.com/vllm-project/vllm) for more options on how to serve your models fast and efficiently.

## 🛜 Evaluate using popular APIs
We support out of the box clients for OpenAI, Anthropic and Google Gemini APIs. First set up your API key:
## 🛜 Evaluate using API

We support how of the box clients for OpenAI, Anthropic and Google Gemini APIs. If you want to evaluate an agent using one of these APIs, you first have to set up your API key in one of two ways:

You can either directly export it:

```bash
export OPENAI_API_KEY=<KEY>
export ANTHROPIC_API_KEY=<KEY>
export GEMINI_API_KEY=<KEY>
```

Then run the evaluation with:
Or you can modify the `SECRETS` file, adding your api keys.

You can then run the evaluation with:

```bash
python eval.py \
agent.type=naive \
agent.max_image_history=0 \
eval.num_workers=64 \
agent.max_history=16 \
eval.num_workers=16 \
client.client_name=openai \
client.model_id=gpt-4o-mini-2024-07-18
```

## Documentation

- [Evaluation Guide](https://github.com/balrog-ai/BALROG/blob/main/docs/evaluation.md) - Detailed instructions for various evaluation scenarios
- [Agent Development](https://github.com/balrog-ai/BALROG/blob/main/docs/agents.md) - Tutorial on creating custom agents
- [Few Shot Learning](https://github.com/balrog-ai/BALROG/blob/main/docs/few_shot_learning.md) - Instructions on how to run Few Shot Learning

We welcome contributions! Please see our [Contributing Guidelines](https://github.com/balrog-ai/BALROG/blob/main/docs/contribution.md) for details.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Citation

If you use BALROG in any of your work, please cite:

```
Expand Down