Skip to content

Commit ace22c3

Browse files
docs(ai): add remote AI worker instructions (#678)
This commit adds documentation about running a AI remote worker to the documentation and updates the orchestrator `get-started` page accordingly. --------- Co-authored-by: Rick Staa <[email protected]>
1 parent 5cf8673 commit ace22c3

File tree

4 files changed

+253
-56
lines changed

4 files changed

+253
-56
lines changed

ai/orchestrators/ai-worker.mdx

+140
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
title: Attach Remote AI Workers
3+
---
4+
5+
## Introduction
6+
7+
The **AI Worker** is a crucial component of the Livepeer AI network, responsible
8+
for performing AI inference tasks. It can be run as a separate process on
9+
compute machines distinct from the Orchestrator or combined with the
10+
Orchestrator on the same machine.
11+
12+
## Key Setup Considerations
13+
14+
- **Startup Configuration**: If you decide to use separate AI Workers, this
15+
**must** be selected at the Orchestrator's startup. A combined Orchestrator
16+
**cannot** simultaneously support remote AI Workers.
17+
- **Shared Configuration File**: Both the Orchestrator and AI Workers use the
18+
`aiModels.json` file (see
19+
[Configuring AI Models](/ai/orchestrators/models-config)).
20+
- The **Orchestrator** uses `aiModels.json` to set model pricing.
21+
- The **AI Worker** uses it to manage the runner containers for each model.
22+
23+
## Remote AI Worker Setup
24+
25+
<Warning>
26+
When using experimental external runner containers, ensure they connect to the
27+
AI Worker and not directly to the Orchestrator.
28+
</Warning>
29+
30+
In a split configuration, the Orchestrator manages multiple AI Workers and
31+
allocates tasks based on the connected workers' capacity. Worker **capacity** is
32+
determined by the following formula:
33+
34+
```
35+
runner container count per pipeline/model_id = capacity per pipeline/model_id
36+
```
37+
38+
The **Orchestrator's capacity** is the sum of the capacities of all connected AI
39+
Workers. This setup enables flexible scaling of compute resources by adding or
40+
removing AI Workers as needed.
41+
42+
### Launch Commands for Remote AI Worker
43+
44+
Below are the launch commands for both the Orchestrator and AI Worker nodes.
45+
46+
<Info>
47+
For the full Orchestrator launch command, see [Start Your AI
48+
Orchestrator](/ai/orchestrators/start-orchestrator).
49+
</Info>
50+
51+
<Accordion title="Show Launch Commands">
52+
53+
#### Orchestrator Command
54+
55+
```bash
56+
docker run \
57+
--name livepeer_ai_orchestrator \
58+
...
59+
-orchestrator \
60+
...
61+
-aiModels /root/.lpData/aiModels.json
62+
```
63+
64+
#### AI Worker Command
65+
66+
```bash
67+
docker run \
68+
--name livepeer_ai_worker \
69+
-aiWorker \
70+
-orchAddr [Orchestrator service address] \
71+
-orchSecret [Orchestrator secret key] \
72+
-nvidia "all" \
73+
-v 6 \
74+
-aiModels /root/.lpData/aiModels.json \
75+
-aiModelsDir /root/.lpData/models \
76+
-testTranscoder=false \
77+
-aiRunnerImage livepeer/ai-runner:latest # Optional
78+
```
79+
80+
<Info>
81+
**Pricing**: Prices in this example may vary and should be adjusted based on
82+
market research and the operational costs of providing compute.
83+
</Info>
84+
85+
</Accordion>
86+
87+
### Configuration Files (`aiModels.json`)
88+
89+
The `aiModels.json` file configures AI model parameters separately for the
90+
Orchestrator and the AI Worker, with each configuration tailored to the specific
91+
needs of that node.
92+
93+
<Info>
94+
For detailed guidance on configuring `aiModels.json` with advanced model
95+
settings, see [Configuring AI Models](/ai/orchestrators/models-config).
96+
</Info>
97+
98+
<Accordion title="Show Configuration Examples">
99+
100+
#### Orchestrator Configuration
101+
102+
```json
103+
[
104+
{
105+
"pipeline": "text-to-image",
106+
"model_id": "SG161222/RealVisXL_V4.0_Lightning",
107+
"price_per_unit": 4768371,
108+
"pixels_per_unit": 1
109+
}
110+
]
111+
```
112+
113+
#### AI Worker Configuration
114+
115+
```json
116+
[
117+
{
118+
"pipeline": "text-to-image",
119+
"model_id": "SG161222/RealVisXL_V4.0_Lightning",
120+
"warm": true,
121+
"optimization_flags": {
122+
"SFAST": true
123+
}
124+
}
125+
]
126+
```
127+
128+
</Accordion>
129+
130+
## Verifying Remote AI Worker Operation
131+
132+
After starting your **remote AI Worker** node, you can verify it is operational
133+
by following the same inference test instructions used for the Orchestrator, as
134+
described in the
135+
[Orchestrator Confirmation Section](/ai/orchestrators/start-orchestrator#confirm-combined-ai-orchestrator-operation).
136+
137+
<Note>
138+
When accessing the AI Runner from a separate machine, replace `localhost` with
139+
the **Worker Node's IP address** in the inference test instructions.
140+
</Note>

ai/orchestrators/models-config.mdx

+13-3
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,24 @@
22
title: Configuring AI Models
33
---
44

5-
Before deploying your AI Orchestrator node on the Livepeer AI network, you must choose the AI models you want to serve for AI inference tasks. This guide assists in configuring these models. The following page, [Download AI Models](/ai/orchestrators/models-download), provides instructions for their download. For details on supported pipelines and models, refer to [Pipelines](/ai/pipelines/text-to-image).
5+
Before deploying your AI Orchestrator node on the Livepeer AI network, you must
6+
choose the AI models you want to serve for AI inference tasks. This guide
7+
assists in configuring these models. The following page,
8+
[Download AI Models](/ai/orchestrators/models-download), provides instructions
9+
for their download. For details on supported pipelines and models, refer to
10+
[Pipelines](/ai/pipelines/text-to-image).
611

712
## Configuration File Format
813

914
Orchestrators specify supported AI models in an `aiModels.json` file, typically
1015
located in the `~/.lpData` directory. Below is an example configuration showing
1116
currently **recommended** models and their respective prices.
1217

18+
<Info>
19+
Pricing used in this example is subject to change and should be set
20+
competitively based on market research and costs to provide the compute.
21+
</Info>
22+
1323
```json
1424
[
1525
{
@@ -94,8 +104,8 @@ currently **recommended** models and their respective prices.
94104

95105
<Note>At this time, these flags are only compatible with **warm** models.</Note>
96106

97-
The Livepeer AI pipelines offer a suite of optimization flags. These
98-
are designed to enhance the performance of **warm** models by either increasing
107+
The Livepeer AI pipelines offer a suite of optimization flags. These are
108+
designed to enhance the performance of **warm** models by either increasing
99109
**inference speed** or reducing **VRAM** usage. Currently, the following flags
100110
are available:
101111

ai/orchestrators/start-orchestrator.mdx

+99-53
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,9 @@ title: Start your AI Orchestrator
33
---
44

55
<Warning>
6-
The Livepeer AI network is currently in its **Beta** stage and is
7-
undergoing active development. Running it on the same machine as your main
8-
Orchestrator or Gateway node may cause stability issues. Please proceed with
9-
caution.
6+
The Livepeer AI network is currently in its **Beta** stage and is undergoing
7+
active development. Running it on the same machine as your main Orchestrator
8+
or Gateway node may cause stability issues. Please proceed with caution.
109
</Warning>
1110

1211
The Livepeer AI network is not yet integrated into the main
@@ -22,9 +21,28 @@ are two methods to run the Livepeer AI network software:
2221
- **Pre-built Binaries**: Pre-built binaries are available if you prefer not to
2322
use Docker.
2423

25-
## Start the AI Orchestrator
24+
## Orchestrator Node Architecture
2625

27-
Please follow the steps below to start your Livepeer AI Orchestrator node:
26+
In the Livepeer AI network, orchestrator operations rely on two primary **node
27+
types**:
28+
29+
- **Orchestrator**: Manages and routes incoming jobs to available compute
30+
resources.
31+
- **Worker**: Performs the actual computation tasks.
32+
33+
The simplest configuration combines both roles on a single machine, utilizing
34+
the machine's GPUs for AI inference tasks, where the orchestrator also functions
35+
as a worker (known as a **combined AI orchestrator**). In this setup, capacity
36+
is limited by the available GPUs and is set as
37+
`runner container count per pipeline/model_id = capacity per pipeline/model_id`.
38+
For expanded scalability, operators can deploy dedicated (remote) worker nodes
39+
that connect to the orchestrator, increasing overall compute capacity.
40+
Instructions for setting up remote workers are available on the
41+
[next page](/ai/orchestrators/ai-worker).
42+
43+
## Start a Combined AI Orchestrator
44+
45+
Please follow the steps below to start your **combined AI orchestrator** node.
2846

2947
<Tabs>
3048
<Tab title="Use Docker (Recommended)">
@@ -49,7 +67,6 @@ Please follow the steps below to start your Livepeer AI Orchestrator node:
4967
```bash
5068
docker pull livepeer/ai-runner:segment-anything-2
5169
```
52-
5370
</Step>
5471
<Step title="Verify the AI Models are Available">
5572
The Livepeer AI network leverages pre-trained AI models for inference tasks. Before launching the AI Orchestrator node, verify that the weights of these models are accessible on your machine. For more information, visit the [Download AI Models](/ai/orchestrators/models-download) page.
@@ -86,7 +103,8 @@ Please follow the steps below to start your Livepeer AI Orchestrator node:
86103
-nvidia "all" \
87104
-aiWorker \
88105
-aiModels /root/.lpData/aiModels.json \
89-
-aiModelsDir ~/.lpData/models
106+
-aiModelsDir ~/.lpData/models \
107+
-aiRunnerImage livepeer/ai-runner:latest # OPTIONAL
90108
```
91109

92110
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
@@ -180,12 +198,11 @@ Please follow the steps below to start your Livepeer AI Orchestrator node:
180198
-nvidia "all" \
181199
-aiWorker \
182200
-aiModels ~/.lpData/aiModels.json \
183-
-aiModelsDir ~/.lpData/models
201+
-aiModelsDir ~/.lpData/models \
202+
-aiRunnerImage livepeer/ai-runner:latest # OPTIONAL
184203
```
185204

186-
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there
187-
188-
are a few **Livepeer AI** specific flags:
205+
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
189206

190207
- `-aiWorker`: This flag enables the AI Worker functionality.
191208
- `-aiModels`: This flag sets the path to the JSON file that contains the AI models.
@@ -214,48 +231,77 @@ are a few **Livepeer AI** specific flags:
214231

215232
</Tabs>
216233

217-
## Confirm the AI Orchestrator is Operational
218-
219-
Once the Livepeer AI Orchestrator node is up and running, validate its
220-
operation by sending an AI inference request directly to the
221-
[ai-runner](https://hub.docker.com/r/livepeer/ai-runner) container. The most
222-
straightforward way to do this is through the
223-
[Swagger UI](https://fastapi.tiangolo.com/features/) interface, accessible at
224-
`http://localhost:8000/docs` if you have loaded the `text-to-image` pipeline.
225-
Note that other pipelines will have different endpoints.
226-
227-
![Swagger UI interface](/images/ai/swagger_ui.png)
228-
229-
<Steps>
230-
<Step title="Access the Swagger UI">
231-
Navigate to `http://localhost:8000/docs` in your web browser to open the Swagger UI interface.
232-
</Step>
233-
<Step title="Initiate an Inference Request">
234-
Initiate an inference request to the `POST /text-to-image` endpoint by clicking the `Try it out` button. Use the following example JSON payload:
235-
236-
```json
237-
{
238-
"prompt": "A cool cat on the beach."
239-
}
240-
```
241-
242-
This request will instruct the AI model to generate an image based on the text in the `prompt` field.
243-
</Step>
244-
<Step title="Inspect the Inference Response">
245-
If the AI Orchestrator node is functioning correctly, you should receive a response similar to the following:
246-
247-
```json
248-
{
249-
"images": [
234+
## Verify Combined AI Orchestrator Operation
235+
236+
Once your **combined Livepeer AI Orchestrator** node is running, verify that the
237+
worker is operational by sending an AI inference request directly to the
238+
[ai-runner](https://hub.docker.com/r/livepeer/ai-runner) container. You can
239+
either use the [Swagger UI](https://fastapi.tiangolo.com/features/) interface or
240+
a `curl` command for this check.
241+
242+
<Tabs>
243+
<Tab title="Use Swagger UI">
244+
<Steps>
245+
<Step title="Access the Swagger UI">
246+
Open your web browser and navigate to `http://localhost:8000/docs` to access the Swagger UI interface.
247+
</Step>
248+
<Step title="Initiate an Inference Request">
249+
In the Swagger UI, locate the `POST /text-to-image` endpoint and click the `Try it out` button. Use the following example JSON payload:
250+
251+
```json
250252
{
251-
"url": "data:image/png;base64,iVBORw0KGgoAA...",
252-
"seed": 2724904334
253+
"prompt": "A cool cat on the beach."
253254
}
254-
]
255-
}
256-
```
255+
```
256+
257+
This request will instruct the AI model to generate an image based on the text in the `prompt` field.
258+
</Step>
259+
<Step title="Inspect the Inference Response">
260+
If the AI Orchestrator node is functioning correctly, you should receive a response similar to the following:
257261

258-
The `url` field contains the base64 encoded image generated by the AI model. To convert this image to a png, use a base64 decoder such as [Base64.guru](https://base64.guru/converter/decode/image/png).
259-
</Step>
262+
```json
263+
{
264+
"images": [
265+
{
266+
"url": "data:image/png;base64,iVBORw0KGgoAA...",
267+
"seed": 2724904334
268+
}
269+
]
270+
}
271+
```
260272

261-
</Steps>
273+
The `url` field contains the base64 encoded image generated by the AI model. To convert this image to PNG, use a base64 decoder such as [Base64.guru](https://base64.guru/converter/decode/image/png).
274+
</Step>
275+
</Steps>
276+
</Tab>
277+
<Tab title="Use curl Command">
278+
<Steps>
279+
<Step title="Send an Inference Request with curl">
280+
Alternatively, you can use the `curl` command to test the AI inference capabilities directly. Run the following command, replacing `<WORKER_NODE_IP>` with the IP address of your worker node:
281+
282+
```bash
283+
curl -X POST "http://localost:8000/text-to-image" -H "Content-Type: application/json" -d '{"prompt": "A cool cat on the beach."}'
284+
```
285+
286+
This sends a POST request to the `text-to-image` endpoint on the worker node with the specified JSON payload.
287+
</Step>
288+
<Step title="Inspect the Response">
289+
If the AI Worker node is functioning correctly, you should receive a response similar to this:
290+
291+
```json
292+
{
293+
"images": [
294+
{
295+
"url": "data:image/png;base64,iVBORw0KGgoAA...",
296+
"seed": 2724904334
297+
}
298+
]
299+
}
300+
```
301+
302+
As with the Swagger UI response, the `url` field contains a base64 encoded image that can be decoded into PNG format using a tool like [Base64.guru](https://base64.guru/converter/decode/image/png).
303+
</Step>
304+
</Steps>
305+
</Tab>
306+
307+
</Tabs>

mint.json

+1
Original file line numberDiff line numberDiff line change
@@ -551,6 +551,7 @@
551551
"ai/orchestrators/models-config",
552552
"ai/orchestrators/models-download",
553553
"ai/orchestrators/start-orchestrator",
554+
"ai/orchestrators/ai-worker",
554555
"ai/orchestrators/benchmarking",
555556
"ai/orchestrators/onchain"
556557
]

0 commit comments

Comments
 (0)