[Example] Add retool tool-calling RL/SFT example (port slime → vime vLLM)

.buildkite/pipeline.yml

@@ -58,7 +58,7 @@ steps:
         python:3.11 bash -c '
           set -euo pipefail
           pip install -q torch --index-url https://download.pytorch.org/whl/cpu
-          pip install -q pytest numpy packaging pyyaml omegaconf tqdm httpx requests ray pybase64 pylatexenc sympy aiohttp pillow safetensors transformers cloudpickle blake3 xxhash zstandard psutil
+          pip install -q pytest numpy packaging pyyaml omegaconf tqdm httpx requests ray pybase64 pylatexenc sympy aiohttp pillow safetensors transformers cloudpickle blake3 xxhash zstandard psutil jinja2
           pip install -q -e . --no-deps
           python tests/test_megatron_argument_validation.py
           python tests/test_value_temperature.py
@@ -85,6 +85,7 @@ steps:
           python tests/test_empty_colocated_weight_bucket.py
           python tests/test_reloadable_process_group_memory_check.py
           python tests/test_ppo_logprob_entropy.py
+          python tests/test_retool_generate.py
           python tests/utils/test_hf_checkpoint_saver.py
         '
 

examples/retool/README.md

@@ -0,0 +1,135 @@
+# Retool: from SFT to RL
+
+This example demonstrates how to use the retool functionality for tool-enabled language model generation.
+
+## Overview
+
+The retool example provides:
+- Safe Python code execution in a sandbox environment
+- Tool registry for managing available tools
+- Integration with language model generation
+- Reward calculation for tool usage
+
+## Files
+
+- `generate_with_retool.py`: Main generation function with tool support
+- `tool_sandbox.py`: Tool execution and safety management
+- `sft_data_processing.py`: Process SFT dataset
+- `rl_data_preprocess.py`: Process the RL (DAPO-Math-17k) dataset
+
+## Usage
+
+1. Setup and download datasets:
+```bash
+cd vime
+pip install -e . --no-deps
+pip install -r examples/retool/requirements.txt
+# For SFT part, you can use later model to RL directly and skip SFT.
+hf download --repo-type dataset JoeYing/ReTool-SFT  --local-dir /root/JoeYing/ReTool-SFT
+hf download Qwen/Qwen3-4B-Instruct-2507 --local-dir /root/Qwen/Qwen3-4B-Instruct-2507
+
+# For RL part
+hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/dapo-math-17k
+hf download --repo-type dataset zhuzilin/aime-2024  --local-dir /root/aime-2024
+# download our SFT model if you want to skip SFT
+hf download font-info/qwen3-4b-sft-SGLang-RL --local-dir /root/font-info/qwen3-4b-sft
+```
+
+2. Create torch dist
+
+Both checkpoints use rope theta `5e6`, which differs from the `1e6` default in
+`scripts/models/qwen3-4B.sh`. Override it with `MODEL_ARGS_ROTARY_BASE` so the
+conversion and the training scripts agree.
+
+For SFT
+```bash
+MODEL_ARGS_ROTARY_BASE=5000000 source scripts/models/qwen3-4B.sh
+PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \
+    ${MODEL_ARGS[@]} \
+    --hf-checkpoint /root/Qwen/Qwen3-4B-Instruct-2507 \
+    --save /root/Qwen/Qwen3-4B-Instruct-2507_torch_dist
+```
+
+Or RL only
+```bash
+MODEL_ARGS_ROTARY_BASE=5000000 source scripts/models/qwen3-4B.sh
+PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \
+    ${MODEL_ARGS[@]} \
+    --hf-checkpoint /root/font-info/qwen3-4b-sft \
+    --save /root/font-info/qwen3-4b-sft_torch_dist
+```
+
+3. SFT:
+```bash
+python examples/retool/sft_data_processing.py
+bash examples/retool/retool_qwen3_4b_sft.sh
+```
+
+4. RL:
+```bash
+bash examples/retool/retool_qwen3_4b_rl.sh
+```
+
+5. Use in your training scripts by importing the generate function:
+```python
+from generate_with_retool import generate, reward_func
+```
+
+The RL script wires these up with:
+```bash
+--custom-generate-function-path generate_with_retool.generate
+--custom-rm-path generate_with_retool.reward_func
+```
+`generate_with_retool` is resolved as a top-level module, so the example
+directory is added to `PYTHONPATH` in the script's Ray runtime env (this is also
+what lets it import its sibling `tool_sandbox`).
+
+## Tool Format
+
+The system uses the following tool format:
+
+```
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{"type": "function", "function": {"name": "code_interpreter", "description": "A tool for executing code.", "parameters": {"type": "object", "properties": {"code": {"type": "string", "description": "The code to execute."}}, "required": ["code"]}}}
+</tools>
+
+For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
+<tool_call>
+{"name": <function-name>, "arguments": <args-json-object>}
+</tool_call>
+```
+
+## Safety Features
+
+- Code execution in isolated sandbox
+- Memory and time limits
+- Dangerous operation detection
+- Allowed module restrictions
+
+Note that `PythonSandbox._check_code_safety` is deliberately strict: it allows
+only the stdlib modules in `PythonSandbox.allowed_modules` (`math`, `random`,
+`statistics`, `decimal`, `fractions`, …) and rejects `eval`/`exec`/`open`,
+dunder access, and imports outside that set. Widen `allowed_modules` if your task
+needs more (e.g. `sympy` or `numpy`).
+
+## Notes on the vLLM port
+
+This example was ported from slime's SGLang implementation. The rollout loop
+talks to vime's vLLM router at `/inference/v1/generate` with a
+`{"model", "token_ids", "sampling_params"}` body, and reads back
+`choices[0].token_ids` plus `choices[0].logprobs.content[i].logprob`.
+
+Two behaviours differ from `vime.rollout.vllm_rollout.generate` on purpose:
+
+- When the engine returns tokens but no usable per-token logprobs, this example
+  marks the sample `ABORTED` instead of substituting zeros. Zero-filled logprobs
+  would desync `rollout_log_probs` from the response tokens and silently corrupt
+  the importance ratio, so the sample is returned to the buffer for retry.
+- The tool-concurrency limit is taken exactly once, inside
+  `ToolRegistry.execute_tool`. `tool_sandbox.SEMAPHORE` is a plain
+  `asyncio.Semaphore` and is not reentrant, so acquiring it in both the caller
+  and the registry needs two permits per tool call and hangs once enough calls
+  are in flight.