Skip to content

Commit b8f8a31

Browse files
luyuzhe111luyuzhe111
authored
doc: improve project README (#4)
1 parent 955b68f commit b8f8a31

2 files changed

Lines changed: 194 additions & 47 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,27 @@ reported the issue. Please try to include as much information as you can. Detail
2020
* Anything unusual about your environment or deployment
2121

2222

23+
## Development Setup
24+
25+
This project uses [uv](https://docs.astral.sh/uv/) for dependency management. Install uv if you haven't already, follow the installation [guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer).
26+
27+
If you're developing or contributing to the `agentcore-rl-toolkit` package itself:
28+
29+
```bash
30+
# Enter the repository
31+
cd agentcore-rl-toolkit
32+
33+
# Create and activate uv environment
34+
uv venv --python 3.13
35+
source .venv/bin/activate
36+
37+
# Install with development dependencies
38+
uv sync --frozen --extra dev
39+
40+
# Install pre-commit hooks
41+
pre-commit install
42+
```
43+
2344
## Contributing via Pull Requests
2445
Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
2546

‎README.md‎

Lines changed: 173 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,167 @@
11

22
# AgentCore RL Toolkit (ART)
33

4-
Toolkit for Seamlessly Enabling RL Training on Any Agent with Bedrock AgentCore.
4+
Toolkit to Seamlessly Enable RL Training on Any Agent with Bedrock AgentCore.
55

6-
## Repo Structure
6+
## Overview
77

8-
- **Main package**: `agentcore-rl-toolkit` is a thin wrapper around of [bedrock-agentcore-sdk-python](https://github.com/aws/bedrock-agentcore-sdk-python/tree/main) that allows developers to start RL training their production agent with only a few lines of code change.
9-
- **Examples**: Located in `examples/` directory, each with their own `pyproject.toml` and dependencies. Their corresponding docker files are located in `.bedrock_agentcore`, most of which have been generated automatically (see [instructions below](#prepare-docker-file)).
8+
**AgentCore RL Toolkit (ART)** is an SDK that helps developers train their LLM agents using reinforcement learning (RL) with **AWS Bedrock AgentCore Runtime (ACR)**. It extends [bedrock-agentcore-sdk-python](https://github.com/aws/bedrock-agentcore-sdk-python/tree/main) to help adapt any existing production agents for RL training with minimal code changes.
109

10+
The toolkit handles the complexity of:
11+
- **Rollout collection**: Automatically captures agent interactions (conversations, tool calls, etc.) during RL training episodes
12+
- **ACR integration**: Works seamlessly with ACR's auto-scaling and session management for efficient parallel rollout generation
13+
- **Data pipeline**: Manages S3 storage and SQS messaging for asynchronous rollout delivery to training engines
14+
- **Reward interface**: Provides a standard base class for implementing custom reward functions
15+
16+
## Installation
17+
18+
```bash
19+
pip install agentcore-rl-toolkit
20+
```
21+
22+
Or with uv:
23+
```bash
24+
uv add agentcore-rl-toolkit
25+
```
26+
27+
To try the example agent applications without installing the package, each example under `examples/` has standalone dependencies. See individual example READMEs (e.g., `examples/strands_math_agent/README.md`).
28+
29+
### What is Bedrock AgentCore Runtime (ACR)?
30+
31+
ACR is AWS's serverless runtime for deploying LLM agents, which can be viewed as **Lambda functions with session continuity**:
32+
33+
- **Session routing**: Requests with the same session ID route to the same container, enabling multi-turn interactions with persistent state
34+
- **Session isolation**: Different session IDs use separate runtime sessions (microVMs), providing strong security isolation between concurrent requests
35+
- **Auto-scaling**: New runtime sessions spin up instantly on demand to handle traffic spikes
36+
- **Sandboxed execution**: Each session runs in a secure microVM environment with resource controls
37+
38+
While session routing is valuable for multi-turn production agents, RL rollouts typically run as single invocations. However, the other properties—**session isolation**, **auto-scaling**, and **sandboxed execution**—still make ACR particularly well-suited for **online RL training**, which requires running many parallel agent rollouts with tool uses securely and efficiently. ACR handles the infrastructure complexity while you focus on agent logic and reward design. After training, you can deploy your fine-tuned model on the same ACR stack with minimal code changes—enabling a fast path from experimentation to production.
39+
40+
### Key Features
41+
42+
- **Minimal migration effort**: Convert your production agent to RL-ready with a simple decorator change (`@app.entrypoint` → `@app.rollout_entrypoint`)
43+
- **Framework support**: Built-in support for Strands framework with extensible architecture for other frameworks
44+
- **Fire-and-forget pattern**: Background async processing eliminates brittle long-lived TCP connections
45+
- **Event-driven architecture**: S3 + SQS integration for reliable, scalable rollout delivery
46+
- **Production-ready**: Direct code reuse from production agents ensures training reflects real deployment behavior
47+
48+
### Starting Point: A Deployment-Ready Agent
49+
50+
This section shows what a deployment-ready ACR agent looks like—use it as a reference if you're new to ACR, or skip ahead if you already have a production agent. The agent must conform to [ACR's HTTP contract](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-http-protocol-contract.html) (endpoints like `/invocations`, `/ping`), but [`BedrockAgentCoreApp`](https://aws.github.io/bedrock-agentcore-starter-toolkit/user-guide/runtime/overview.html) handles this for you.
51+
52+
For Strands agents, see the [Deploy to Bedrock AgentCore guide](https://strandsagents.com/latest/documentation/docs/user-guide/deploy/deploy_to_bedrock_agentcore/python/). For other frameworks, see the [ACR Getting Started documentation](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-getting-started.html).
53+
54+
A minimal Strands agent looks like (see [`basic_app.py`](examples/strands_math_agent/basic_app.py) for full example):
55+
56+
```python
57+
from bedrock_agentcore.runtime import BedrockAgentCoreApp
58+
from strands import Agent
59+
from strands.models import BedrockModel
60+
61+
app = BedrockAgentCoreApp()
62+
model = BedrockModel(model_id="us.anthropic.claude-sonnet-4-20250514-v1:0")
63+
agent = Agent(model=model, tools=[...], system_prompt="...")
64+
65+
@app.entrypoint
66+
async def invoke_agent(payload):
67+
response = await agent.invoke_async(payload.get("prompt"))
68+
return response.message["content"][0]["text"]
69+
70+
if __name__ == "__main__":
71+
app.run()
72+
```
73+
74+
### Adapting for RL Training
75+
76+
Starting from a deployment-ready agent like above, here are the changes needed for RL training (see [`rl_app.py`](examples/strands_math_agent/rl_app.py) for full example):
77+
```python
78+
from agentcore_rl_toolkit import StrandsAgentCoreRLApp, StrandsRolloutCollector
79+
from strands import Agent
80+
81+
app = StrandsAgentCoreRLApp()
82+
model = app.create_openai_compatible_model()
83+
rollout_collector = StrandsRolloutCollector()
84+
agent = Agent(model=model, system_prompt="...", hooks=[rollout_collector])
85+
reward_fn = GSM8KReward() # user-defined reward function
86+
87+
@app.rollout_entrypoint
88+
async def invoke_agent(payload):
89+
response = await agent.invoke_async(payload.get("prompt"))
90+
rollout_data = rollout_collector.get_rollout_data()
91+
rewards = reward_fn(response_text=response.message["content"][0]["text"], ground_truth=payload.get("answer"))
92+
return {"rollout_data": rollout_data, "rewards": rewards}
93+
```
94+
95+
**Key changes:**
96+
1. `BedrockAgentCoreApp` → `StrandsAgentCoreRLApp`
97+
2. `BedrockModel` → `app.create_openai_compatible_model()` (points to training cluster)
98+
3. Add `StrandsRolloutCollector` hook to capture conversations
99+
4. `@app.entrypoint` → `@app.rollout_entrypoint`
100+
5. Return `{"rollout_data": ..., "rewards": ...}` instead of text
11101

12102
## Start Training on Example Agents
13-
AgentCore runtime is currently supported by the following training library.
14-
- [verl](https://github.com/volcengine/verl): related [PR](https://github.com/volcengine/verl/pull/4216).
15103

16-
Before training, build the docker for the RL-ready application and upload to ECR. To do this, follow the steps below:
104+
### High-Level Training Architecture
105+
106+
The training architecture follows a **decoupled design** where agent rollouts and the training engine run separately:
107+
108+
```
109+
┌────────────────────────────────────────────────────────────────┐
110+
│ Training Cluster │
111+
│ ┌─────────────────────┐ ┌─────────────────────┐ │
112+
│ │ Training Engine ├───►│ Inference Servers │ │
113+
│ │ (e.g., veRL) │ │ (vLLM/SGLang) │ │
114+
│ └───────┬─────────────┘ └───────▲─────────────┘ │
115+
│ │ │ │
116+
└───────────────┼──────────────────────────┼─────────────────────┘
117+
│ 1. Submit N prompts │ 2. Model inference
118+
│ to ACR │ calls from agents
119+
▼ │
120+
┌────────────────────────────────────────────────────────────────┐
121+
│ AWS Bedrock AgentCore Runtime │
122+
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
123+
│ │ Agent │ │ Agent │ ... │ Agent │ │
124+
│ │ Session 1 │ │ Session 2 │ │ Session N │ │
125+
│ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
126+
│ │ │ │ │
127+
│ └──────────────┴───────────────────┘ │
128+
│ │ 3. Save rollouts + rewards │
129+
│ ▼ │
130+
│ ┌─────────────────┐ ┌─────────────────┐ │
131+
│ │ S3 (rollouts) │───►│ SQS (notify) │ │
132+
│ └─────────────────┘ └────────┬────────┘ │
133+
└───────────────────────────────────────────┼────────────────────┘
134+
│ 4. Poll for results
135+
▼
136+
Training Engine receives
137+
rollouts for policy update
138+
```
139+
140+
**Workflow:**
141+
1. **Prompt submission**: Training engine submits N prompts to ACR, which auto-scales to spin up N parallel agent sessions
142+
2. **Agent execution**: Each agent session processes its prompt, calling the inference server for model responses
143+
3. **Rollout collection**: Agents save rollouts and rewards to S3, send notifications via SQS
144+
4. **Policy update**: Training engine polls SQS, collects rollouts from S3, and updates the model
145+
146+
This architecture enables parallel and highly efficient rollouts with secure execution during RL training. The decoupled design means training libraries only need the agent's container image to start training—agent code and dependencies stay completely separate from the training library.
147+
148+
### Supported Training Libraries
149+
150+
AgentCore runtime is currently supported by:
151+
- **[veRL](https://github.com/volcengine/verl)**: See the integration [PR](https://github.com/volcengine/verl/pull/4216).
152+
153+
### Prepare Your Agent Container
154+
155+
ACR deploys agents as Docker containers. Most Dockerfiles can be auto-generated with the [AgentCore CLI](https://aws.github.io/bedrock-agentcore-starter-toolkit/api-reference/cli.html). Example command:
156+
157+
```bash
158+
# Run from project root
159+
agentcore configure --entrypoint examples/strands_math_agent/rl_app.py \
160+
--requirements-file examples/strands_math_agent/pyproject.toml \
161+
--deployment-type container --disable-memory --non-interactive
162+
```
163+
164+
You can further customize these files if needed. Pre-generated Dockerfiles for all examples are provided in `.bedrock_agentcore/`. Once you have a Dockerfile, follow these steps to build and push your agent image to ECR.
17165

18166
### Setup Credentials and Environment Variables
19167

@@ -38,56 +186,34 @@ chmod +x scripts/build_docker_image_and_push_to_ecr.sh
38186
bash ./scripts/build_docker_image_and_push_to_ecr.sh --dockerfile=.bedrock_agentcore/examples_strands_math_agent_rl_app/Dockerfile --tag=dev
39187
```
40188

41-
Then, go to the training library of your choice and simply provide agentcore specific config args to start training.
42-
43-
44-
## Development
189+
### Start Training with veRL
45190

46-
### Installation
47-
48-
This project uses [uv](https://docs.astral.sh/uv/) for dependency management. Install uv if you haven't already, follow the installation [guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) here.
49-
50-
### For Package Development
51-
52-
If you're developing or contributing to the `agentcore-rl-toolkit` package itself:
191+
Once your container is pushed to ECR, configure veRL with the AgentCore-specific parameters:
53192

54193
```bash
55-
# Enter the repository
56-
cd agentcore-rl-toolkit
57-
58-
# Create and activate uv environment
59-
uv venv --python 3.13
60-
source .venv/bin/activate
61-
62-
# Install with development dependencies
63-
uv sync --frozen --extra dev
64-
65-
# Install pre-commit hooks
66-
pre-commit install
194+
# AgentCore configuration for veRL
195+
actor_rollout_ref.rollout.name=agentcore
196+
actor_rollout_ref.rollout.agentcore.agent_name=<your-agent-name>
197+
actor_rollout_ref.rollout.agentcore.container_uri=<account>.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>
198+
actor_rollout_ref.rollout.agentcore.s3_bucket=<your-s3-bucket>
67199
```
68200

69-
Additionally, when co-developing the toolkit together with examples, add the following to the example app's docker file so that changes to the toolkit is reflected in the container.
201+
| Parameter | Description |
202+
|-----------|-------------|
203+
| `agent_name` | Name for your agent in ACR |
204+
| `container_uri` | ECR URI of your RL-ready agent container |
205+
| `s3_bucket` | S3 bucket for storing rollout data |
70206

71-
```bash
72-
COPY . .
73-
RUN uv pip install --force-reinstall --no-deps .
74-
```
75-
76-
### For Running Examples
77-
78-
Each example has its own dependencies and can be installed independently. Follow the README for specific examples there (e.g., `examples/strands_math_agent/README.md`).
207+
For a complete example, see the [veRL AgentCore integration PR](https://github.com/volcengine/verl/pull/4216).
79208

80-
## Appendix
209+
## Contributing
81210

82-
### Prepare Docker file
211+
We welcome contributions!
83212

84-
Docker file for most examples can be automatically generated with the `agentcore` CLI. Use `examples/strands_math_agent` as an example:
85-
86-
```bash
87-
agentcore configure --entrypoint examples/strands_math_agent/rl_app.py --requirements-file examples/strands_math_agent/pyproject.toml --deployment-type container --disable-memory --non-interactive
88-
```
213+
- **Adding examples**: Create a new folder under `examples/` with its own `pyproject.toml`
214+
- **Improving the core library**: See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup
89215

90-
Make sure to run the command in project root.
216+
For bug reports, feature requests, and pull request guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).
91217

92218
## Security
93219

0 commit comments

Comments
 (0)