Skip to content

Commit d92b839

Browse files
lyzustclyzustc
authored
docs(verl): add verl training setup and launch instructions (#73)
* docs(verl): add verl training setup and launch instructions * docs(verl): change verl environment installation to using pip + pyproject.toml and megatron commands
1 parent 3a35cbb commit d92b839

4 files changed

Lines changed: 217 additions & 5 deletions

File tree

‎docs/site/astro.config.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ export default defineConfig({
4646
items: [
4747
{ label: 'slime', slug: 'guides/slime-backend-setup' },
4848
{ label: 'rllm', slug: 'guides/rllm-backend-setup' },
49-
{ label: 'verl (coming soon)', slug: 'guides/verl-backend-setup' },
49+
{ label: 'verl', slug: 'guides/verl-backend-setup' },
5050
],
5151
},
5252
],

‎docs/site/src/content/docs/guides/verl-backend-setup.md‎

Lines changed: 153 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,159 @@
11
---
22
title: verl backend setup
3-
description: Runbook for training with verl (self-hosted FSDP + vLLM/SGLang).
3+
description: Train an AgentCore Runtime-deployed agent with the verl training backend (Megatron / FSDP + vLLM).
44
---
55

6-
:::note[Coming Soon]
7-
The verl training backend integration is coming soon. This page will be updated once the integration is available.
6+
This doc describes how to train an AgentCore Runtime-deployed agent with the [verl](https://github.com/volcengine/verl) training backend. Note that this is the direct integration with official verl, instead of through other packages like [rllm](https://awslabs.github.io/agentcore-rl-toolkit/guides/rllm-backend-setup/). We implement a thin AgentCore layer on top of official verl's PPO trainer, launched directly via `python -m agentcore_rl_toolkit.backends.verl.main`.
7+
8+
## Prerequisites
9+
10+
- A GPU cluster with **CUDA>=12.8** installed.
11+
- Python 3.12+ and [`uv`](https://docs.astral.sh/uv/).
12+
- AWS credentials with permission to invoke an AgentCore Runtime and read/write an S3 bucket.
13+
- An AgentCore Runtime deployment of your agent — follow the [Prepare agent for RL](/agentcore-rl-toolkit/guides/agent-adaptation/) guide. Save the resulting **runtime ARN** — required as `actor_rollout_ref.rollout.agentcore.agent_runtime_arn` below for launching agent rollout sessions.
14+
- An S3 bucket for rollout result delivery — required as `actor_rollout_ref.rollout.agentcore.s3_bucket` below for acquiring rewards.
15+
16+
## Installation
17+
18+
The verl backend has a heavyweight dependency stack (vLLM, Megatron-Core, Megatron-Bridge, Transformer Engine, Apex, flash-attn). Run the commands below to install environment for verl 0.8.0.
19+
20+
:::note
21+
The following commands assume CUDA 13.0 is installed at `/usr/local/cuda-13.0`; adjust `CUDA_HOME` and `--torch-backend` if yours differs.
822
:::
923

10-
In the meantime, verl is also available as a training backend via [rllm](/agentcore-rl-toolkit/guides/rllm-backend-setup/). See the rllm backend setup guide for instructions on running training with verl through rllm.
24+
```bash
25+
export CUDA_HOME=/usr/local/cuda-13.0
26+
uv pip install -e .[verl] --torch-backend=cu130
27+
bash src/agentcore_rl_toolkit/backends/verl/scripts/install_megatron.sh cu130
28+
```
29+
30+
## Prepare data
31+
32+
Verl reads training and validation data from parquet files. We make each training example (one row in parquet file) in the dataset processed and forwarded as a raw python dict in the `payload` to the AgentCore Runtime session — no trainer-side tokenization. So you just need to put every field that your implemented agent takes as input from session's `payload` as a top-level column of dataset parquet file.
33+
34+
For a reference example, we provide the data preprocessing script ([`preprocess_gsm8k.py`](https://github.com/awslabs/agentcore-rl-toolkit/blob/main/src/agentcore_rl_toolkit/backends/verl/examples/math_agent/preprocess_gsm8k.py)) used for GSM8K dataset in [math agent](https://github.com/awslabs/agentcore-rl-toolkit/blob/main/examples/strands_math_agent/rl_app.py). By running it, you downloads `openai/gsm8k`, extracts the gold answer from the `#### N` marker, and writes two Parquet files:
35+
36+
```bash
37+
cd src/agentcore_rl_toolkit/backends/verl/examples/math_agent
38+
python preprocess_gsm8k.py --output-dir gsm8k
39+
```
40+
41+
Each row in the parquet file has two columns:
42+
43+
| Column | Purpose |
44+
|--------|---------|
45+
| `prompt` | The question text. Reaches the agent as `payload["prompt"]`. |
46+
| `answer` | The ground-truth final answer. Reaches the agent as `payload["answer"]` and is passed to `GSM8KReward` as `ground_truth`. |
47+
48+
To train your own task, write a script that produces a Parquet file with whatever columns your agent's `payload` expects.
49+
50+
## Training Configuration
51+
52+
Verl uses yaml for training configuration, see verl's [official doc](https://verl.readthedocs.io/en/latest/examples/config.html) for configuration explanation. We make a training config file at [`src/agentcore_rl_toolkit/backends/verl/config/agentcore_grpo.yaml`](https://github.com/awslabs/agentcore-rl-toolkit/blob/main/src/agentcore_rl_toolkit/backends/verl/config/agentcore_grpo.yaml). It inherits verl's full training config and adds the following new arguments:
53+
54+
```yaml
55+
actor_rollout_ref:
56+
rollout:
57+
agentcore:
58+
agent_runtime_arn: "" # REQUIRED — The ARN of your deployed agent at AgentCore Runtime
59+
s3_bucket: "" # REQUIRED — S3 bucket for saving rewards and other artifacts of agent rollouts
60+
reqs_per_sec: 25 # AgentCore Runtime invoke TPS limit (default 25, per-account)
61+
max_pool_connections: 10 # boto3 connection pool size (peak simultaneous HTTP calls of AgentCore invoke and S3 poll)
62+
max_rollout_time: 1800 # Max running time of an AgentCore Runtime session in seconds
63+
gateway_port: 9090 # local model-gateway port
64+
gateway_store: memory # gateway trace store backend
65+
gateway_cumulative_token_mode: false # Turn on model gateway's cumulative token mode or not
66+
gateway_renderer_model_family: auto # Renderer family used for model gateway's cumulative token mode
67+
68+
actor:
69+
ppo_mini_steps: 1 # The number of ppo mini steps per global training step
70+
```
71+
72+
Please see comments in [`agentcore_grpo.yaml`](https://github.com/awslabs/agentcore-rl-toolkit/blob/main/src/agentcore_rl_toolkit/backends/verl/config/agentcore_grpo.yaml) for more instructions.
73+
74+
You can edit these values directly in the yaml file, or as the next section's example script does, override them on the command line of training launch.
75+
76+
## Launch training
77+
78+
Training is launched with `python -m agentcore_rl_toolkit.backends.verl.main`, which loads `agentcore_grpo.yaml` and accepts standard yaml overrides from command lines.
79+
80+
We use math agent as a reference example, see complete training script at [`src/agentcore_rl_toolkit/backends/verl/examples/math_agent/run_agentcore_grpo.sh`](https://github.com/awslabs/agentcore-rl-toolkit/blob/main/src/agentcore_rl_toolkit/backends/verl/examples/math_agent/run_agentcore_grpo.sh).
81+
82+
Before running the training:
83+
84+
1. Deploy the math agent RL app to AgentCore Runtime following the [guide](https://github.com/awslabs/agentcore-rl-toolkit/blob/main/examples/strands_math_agent/README.md).
85+
2. Get the Bedrock AgentCore Runtime ARN of your deployed agent, and create a S3 bucket for saving rollout rewards.
86+
3. Get your wandb API key for training curve visualization (set `trainer.logger` below to use other visualization platform).
87+
88+
The training can be launched with the following commands:
89+
90+
```bash
91+
export VLLM_ALLREDUCE_USE_SYMM_MEM=0
92+
export CUDA_DEVICE_MAX_CONNECTIONS=1
93+
export CUDA_HOME=/usr/local/cuda-13.0
94+
VENV_CU13_LIB=$(python -c "import sysconfig, os; print(os.path.join(sysconfig.get_path('purelib'), 'nvidia', 'cu13', 'lib'))")
95+
export LD_LIBRARY_PATH=$VENV_CU13_LIB:$LD_LIBRARY_PATH
96+
export WANDB_API_KEY="your-wandb-api-key"
97+
98+
python3 -m agentcore_rl_toolkit.backends.verl.main \
99+
model_engine=megatron \
100+
algorithm.adv_estimator=grpo \
101+
data.train_files="$gsm8k/gsm8k_agent_train.parquet" \
102+
data.val_files="$gsm8k/gsm8k_agent_test.parquet" \
103+
data.train_batch_size=64 \
104+
data.val_batch_size=256 \
105+
data.max_prompt_length=14336 \
106+
data.max_response_length=2048 \
107+
actor_rollout_ref.model.path=Qwen/Qwen3-4B-Instruct-2507 \
108+
actor_rollout_ref.model.lora.rank=128 \
109+
actor_rollout_ref.model.lora.alpha=256 \
110+
actor_rollout_ref.model.lora.merge=true \
111+
actor_rollout_ref.actor.optim.lr=1e-5 \
112+
actor_rollout_ref.actor.ppo_mini_steps=1 \
113+
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \
114+
actor_rollout_ref.actor.megatron.pipeline_model_parallel_size=1 \
115+
actor_rollout_ref.actor.megatron.tensor_model_parallel_size=2 \
116+
actor_rollout_ref.actor.megatron.use_dist_checkpointing=False \
117+
actor_rollout_ref.actor.megatron.use_mbridge=True \
118+
actor_rollout_ref.actor.megatron.vanilla_mbridge=False \
119+
actor_rollout_ref.actor.use_kl_loss=True \
120+
actor_rollout_ref.actor.kl_loss_coef=0.001 \
121+
actor_rollout_ref.actor.kl_loss_type=low_var_kl \
122+
actor_rollout_ref.actor.entropy_coeff=0 \
123+
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=1 \
124+
actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
125+
actor_rollout_ref.rollout.name=vllm \
126+
actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
127+
actor_rollout_ref.rollout.n=4 \
128+
+actor_rollout_ref.rollout.engine_kwargs.vllm.enable_auto_tool_choice=true \
129+
+actor_rollout_ref.rollout.engine_kwargs.vllm.tool_call_parser=hermes \
130+
actor_rollout_ref.rollout.agentcore.agent_runtime_arn=your-math-agent-arn \
131+
actor_rollout_ref.rollout.agentcore.s3_bucket=your-s3-bucket \
132+
actor_rollout_ref.rollout.agentcore.max_rollout_time=180 \
133+
actor_rollout_ref.rollout.agentcore.gateway_cumulative_token_mode=true \
134+
actor_rollout_ref.rollout.agentcore.gateway_renderer_model_family=qwen3 \
135+
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=1 \
136+
actor_rollout_ref.ref.megatron.pipeline_model_parallel_size=1 \
137+
actor_rollout_ref.ref.megatron.tensor_model_parallel_size=2 \
138+
algorithm.use_kl_in_reward=False \
139+
trainer.critic_warmup=0 \
140+
trainer.default_local_dir=exp_agentcore_grpo \
141+
trainer.logger='["console","wandb"]' \
142+
trainer.project_name='agentcore-rl-toolkit' \
143+
trainer.experiment_name='gsm8k' \
144+
trainer.n_gpus_per_node=8 \
145+
trainer.nnodes=1 \
146+
trainer.save_freq=100 \
147+
trainer.test_freq=10 \
148+
trainer.val_before_train=true \
149+
trainer.total_epochs=1
150+
```
151+
152+
Things worth noting:
153+
154+
- **`model_engine`** — `megatron` (used in the example) or `dp` for FSDP.
155+
- **`ppo_mini_steps`** — This is specific to our verl + AgentCore integration. One agent rollout trajectory can expand into multiple sequences, so the number of PPO mini-batches per global step isn't fixed; this pins it explicitly.
156+
- **`enable_auto_tool_choice` / `tool_call_parser=hermes`** — required for tool-calling agents (the math agent uses a calculator tool). Match the parser to your model family.
157+
- **Tool-call + cumulative tokens** — the example enables gateway's cumulative token mode by setting `gateway_cumulative_token_mode=true` with `gateway_renderer_model_family=qwen3`. See this [PR](https://github.com/rllm-org/rllm/pull/596) for more information about it.
158+
159+
For the full list of tunable fields, consult verl's yaml config [documentation](https://verl.readthedocs.io/en/latest/examples/config.html).

‎pyproject.toml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,10 +52,32 @@ rllm = [
5252
# in agentcore-rl-toolkit, which would create a circular dependency.
5353
"rllm @ git+https://github.com/rllm-org/rllm.git",
5454
]
55+
verl = [
56+
"transformers>=5.5.3",
57+
"numpy",
58+
"verl==0.8.0",
59+
"ray",
60+
"torch>=2.10.0",
61+
"torchvision>=0.23.0",
62+
"vllm==0.21.0",
63+
"flash-attn==2.8.3",
64+
"qwen-vl-utils",
65+
"rllm-model-gateway @ git+https://github.com/rllm-org/rllm.git#subdirectory=rllm-model-gateway",
66+
]
5567

5668
[project.urls]
5769
Homepage = "https://github.com/awslabs/agentcore-rl-toolkit"
5870

71+
[tool.uv]
72+
override-dependencies = [
73+
"numpy>=1.26.0",
74+
]
75+
76+
[tool.uv.extra-build-dependencies]
77+
flash-attn = [
78+
{ requirement = "torch", match-runtime = true },
79+
]
80+
5981
[tool.hatch.metadata]
6082
# Required so the [rllm] optional extra can pin rllm via a git URL
6183
# (PEP 508 direct reference). Hatchling rejects these by default.

‎src/agentcore_rl_toolkit/backends/verl/scripts/install_megatron.sh‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
# Install Megatron-related dependencies for verl.
3+
#
4+
# These packages require special build flags that cannot be expressed
5+
# in pyproject.toml. Run this AFTER installing the verl backend:
6+
# uv pip install -e ".[verl]" --torch-backend=<cu128|cu129|cu130|...>
7+
#
8+
# Usage:
9+
# bash src/agentcore_rl_toolkit/backends/verl/scripts/install_megatron.sh <cu128|cu129|cu130|...>
10+
11+
set -euo pipefail
12+
13+
TORCH_BACKEND="${1:?Usage: bash scripts/install_megatron.sh <torch-backend, e.g. cu128 or cu129>}"
14+
echo "=== Megatron dependency installer for rLLM ==="
15+
echo "TORCH_BACKEND=${TORCH_BACKEND}"
16+
17+
CUDA_MAJOR="${TORCH_BACKEND#cu}"; CUDA_MAJOR="${CUDA_MAJOR:0:2}" # e.g. cu130 -> 13
18+
19+
echo "[1/5] Installing nvidia-modelopt..."
20+
uv pip install 'nvidia-modelopt>=0.37.0'
21+
22+
echo "[2/5] Installing transformer-engine (this may take a while)..."
23+
MAX_JOBS=128 uv pip install --no-cache --no-build-isolation \
24+
"transformer_engine[pytorch,core-cu${CUDA_MAJOR}]==2.11"
25+
26+
# megatron-core > 0.15.0 required for numpy>=2.0.0 compatibility
27+
echo "[3/5] Installing megatron-core..."
28+
uv pip install --no-deps megatron-core==0.17.1
29+
30+
echo "[4/5] Installing megatron-bridge..."
31+
# Pinned to 691a377f (2026-05-19): "Add external trainer integration helpers (#3813)"
32+
# verl 0.8.0 requires LinearForLastLayer which was added in this commit (unreleased, post-v0.4.2).
33+
uv pip install --no-deps git+https://github.com/NVIDIA-NeMo/Megatron-Bridge.git@691a377f
34+
35+
echo "[5/5] Installing NVIDIA Apex (required for gradient accumulation fusion)..."
36+
APEX_PARALLEL_BUILD=8 APEX_CPP_EXT=1 APEX_CUDA_EXT=1 \
37+
uv pip install -v --no-cache --no-build-isolation \
38+
git+https://github.com/NVIDIA/apex.git --torch-backend="${TORCH_BACKEND}"
39+
40+
echo ""
41+
echo "=== Megatron dependencies installed successfully ==="

0 commit comments

Comments
 (0)