You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(ai): add remote AI worker instructions (#678)
This commit adds documentation about running a AI remote worker to the documentation and updates the orchestrator `get-started` page accordingly.
---------
Co-authored-by: Rick Staa <[email protected]>
Copy file name to clipboardexpand all lines: ai/orchestrators/models-config.mdx
+13-3
Original file line number
Diff line number
Diff line change
@@ -2,14 +2,24 @@
2
2
title: Configuring AI Models
3
3
---
4
4
5
-
Before deploying your AI Orchestrator node on the Livepeer AI network, you must choose the AI models you want to serve for AI inference tasks. This guide assists in configuring these models. The following page, [Download AI Models](/ai/orchestrators/models-download), provides instructions for their download. For details on supported pipelines and models, refer to [Pipelines](/ai/pipelines/text-to-image).
5
+
Before deploying your AI Orchestrator node on the Livepeer AI network, you must
6
+
choose the AI models you want to serve for AI inference tasks. This guide
7
+
assists in configuring these models. The following page,
8
+
[Download AI Models](/ai/orchestrators/models-download), provides instructions
9
+
for their download. For details on supported pipelines and models, refer to
10
+
[Pipelines](/ai/pipelines/text-to-image).
6
11
7
12
## Configuration File Format
8
13
9
14
Orchestrators specify supported AI models in an `aiModels.json` file, typically
10
15
located in the `~/.lpData` directory. Below is an example configuration showing
11
16
currently **recommended** models and their respective prices.
12
17
18
+
<Info>
19
+
Pricing used in this example is subject to change and should be set
20
+
competitively based on market research and costs to provide the compute.
21
+
</Info>
22
+
13
23
```json
14
24
[
15
25
{
@@ -94,8 +104,8 @@ currently **recommended** models and their respective prices.
94
104
95
105
<Note>At this time, these flags are only compatible with **warm** models.</Note>
96
106
97
-
The Livepeer AI pipelines offer a suite of optimization flags. These
98
-
are designed to enhance the performance of **warm** models by either increasing
107
+
The Livepeer AI pipelines offer a suite of optimization flags. These are
108
+
designed to enhance the performance of **warm** models by either increasing
99
109
**inference speed** or reducing **VRAM** usage. Currently, the following flags
Copy file name to clipboardexpand all lines: ai/orchestrators/start-orchestrator.mdx
+99-53
Original file line number
Diff line number
Diff line change
@@ -3,10 +3,9 @@ title: Start your AI Orchestrator
3
3
---
4
4
5
5
<Warning>
6
-
The Livepeer AI network is currently in its **Beta** stage and is
7
-
undergoing active development. Running it on the same machine as your main
8
-
Orchestrator or Gateway node may cause stability issues. Please proceed with
9
-
caution.
6
+
The Livepeer AI network is currently in its **Beta** stage and is undergoing
7
+
active development. Running it on the same machine as your main Orchestrator
8
+
or Gateway node may cause stability issues. Please proceed with caution.
10
9
</Warning>
11
10
12
11
The Livepeer AI network is not yet integrated into the main
@@ -22,9 +21,28 @@ are two methods to run the Livepeer AI network software:
22
21
-**Pre-built Binaries**: Pre-built binaries are available if you prefer not to
23
22
use Docker.
24
23
25
-
## Start the AI Orchestrator
24
+
## Orchestrator Node Architecture
26
25
27
-
Please follow the steps below to start your Livepeer AI Orchestrator node:
26
+
In the Livepeer AI network, orchestrator operations rely on two primary **node
27
+
types**:
28
+
29
+
-**Orchestrator**: Manages and routes incoming jobs to available compute
30
+
resources.
31
+
-**Worker**: Performs the actual computation tasks.
32
+
33
+
The simplest configuration combines both roles on a single machine, utilizing
34
+
the machine's GPUs for AI inference tasks, where the orchestrator also functions
35
+
as a worker (known as a **combined AI orchestrator**). In this setup, capacity
36
+
is limited by the available GPUs and is set as
37
+
`runner container count per pipeline/model_id = capacity per pipeline/model_id`.
38
+
For expanded scalability, operators can deploy dedicated (remote) worker nodes
39
+
that connect to the orchestrator, increasing overall compute capacity.
40
+
Instructions for setting up remote workers are available on the
41
+
[next page](/ai/orchestrators/ai-worker).
42
+
43
+
## Start a Combined AI Orchestrator
44
+
45
+
Please follow the steps below to start your **combined AI orchestrator** node.
28
46
29
47
<Tabs>
30
48
<Tabtitle="Use Docker (Recommended)">
@@ -49,7 +67,6 @@ Please follow the steps below to start your Livepeer AI Orchestrator node:
49
67
```bash
50
68
docker pull livepeer/ai-runner:segment-anything-2
51
69
```
52
-
53
70
</Step>
54
71
<Steptitle="Verify the AI Models are Available">
55
72
The Livepeer AI network leverages pre-trained AI models for inference tasks. Before launching the AI Orchestrator node, verify that the weights of these models are accessible on your machine. For more information, visit the [Download AI Models](/ai/orchestrators/models-download) page.
@@ -86,7 +103,8 @@ Please follow the steps below to start your Livepeer AI Orchestrator node:
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
@@ -180,12 +198,11 @@ Please follow the steps below to start your Livepeer AI Orchestrator node:
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there
187
-
188
-
are a few **Livepeer AI** specific flags:
205
+
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
189
206
190
207
-`-aiWorker`: This flag enables the AI Worker functionality.
191
208
- `-aiModels`: This flag sets the path to the JSON file that contains the AI models.
@@ -214,48 +231,77 @@ are a few **Livepeer AI** specific flags:
214
231
215
232
</Tabs>
216
233
217
-
## Confirm the AI Orchestrator is Operational
218
-
219
-
Once the Livepeer AI Orchestrator node is up and running, validate its
220
-
operation by sending an AI inference request directly to the
221
-
[ai-runner](https://hub.docker.com/r/livepeer/ai-runner) container. The most
222
-
straightforward way to do this is through the
223
-
[Swagger UI](https://fastapi.tiangolo.com/features/) interface, accessible at
224
-
`http://localhost:8000/docs` if you have loaded the `text-to-image` pipeline.
225
-
Note that other pipelines will have different endpoints.
Navigate to `http://localhost:8000/docs` in your web browser to open the Swagger UI interface.
232
-
</Step>
233
-
<Steptitle="Initiate an Inference Request">
234
-
Initiate an inference request to the `POST /text-to-image` endpoint by clicking the `Try it out` button. Use the following example JSON payload:
235
-
236
-
```json
237
-
{
238
-
"prompt": "A cool cat on the beach."
239
-
}
240
-
```
241
-
242
-
This request will instruct the AI model to generate an image based on the text in the `prompt` field.
243
-
</Step>
244
-
<Steptitle="Inspect the Inference Response">
245
-
If the AI Orchestrator node is functioning correctly, you should receive a response similar to the following:
246
-
247
-
```json
248
-
{
249
-
"images": [
234
+
## Verify Combined AI Orchestrator Operation
235
+
236
+
Once your **combined Livepeer AI Orchestrator** node is running, verify that the
237
+
worker is operational by sending an AI inference request directly to the
238
+
[ai-runner](https://hub.docker.com/r/livepeer/ai-runner) container. You can
239
+
either use the [Swagger UI](https://fastapi.tiangolo.com/features/) interface or
240
+
a `curl` command for this check.
241
+
242
+
<Tabs>
243
+
<Tabtitle="Use Swagger UI">
244
+
<Steps>
245
+
<Steptitle="Access the Swagger UI">
246
+
Open your web browser and navigate to `http://localhost:8000/docs` to access the Swagger UI interface.
247
+
</Step>
248
+
<Steptitle="Initiate an Inference Request">
249
+
In the Swagger UI, locate the `POST /text-to-image` endpoint and click the `Try it out` button. Use the following example JSON payload:
250
+
251
+
```json
250
252
{
251
-
"url": "data:image/png;base64,iVBORw0KGgoAA...",
252
-
"seed": 2724904334
253
+
"prompt": "A cool cat on the beach."
253
254
}
254
-
]
255
-
}
256
-
```
255
+
```
256
+
257
+
This request will instruct the AI model to generate an image based on the text in the `prompt` field.
258
+
</Step>
259
+
<Steptitle="Inspect the Inference Response">
260
+
If the AI Orchestrator node is functioning correctly, you should receive a response similar to the following:
257
261
258
-
The `url` field contains the base64 encoded image generated by the AI model. To convert this image to a png, use a base64 decoder such as [Base64.guru](https://base64.guru/converter/decode/image/png).
259
-
</Step>
262
+
```json
263
+
{
264
+
"images": [
265
+
{
266
+
"url": "data:image/png;base64,iVBORw0KGgoAA...",
267
+
"seed": 2724904334
268
+
}
269
+
]
270
+
}
271
+
```
260
272
261
-
</Steps>
273
+
The `url` field contains the base64 encoded image generated by the AI model. To convert this image to PNG, use a base64 decoder such as [Base64.guru](https://base64.guru/converter/decode/image/png).
274
+
</Step>
275
+
</Steps>
276
+
</Tab>
277
+
<Tabtitle="Use curl Command">
278
+
<Steps>
279
+
<Steptitle="Send an Inference Request with curl">
280
+
Alternatively, you can use the `curl` command to test the AI inference capabilities directly. Run the following command, replacing `<WORKER_NODE_IP>` with the IP address of your worker node:
281
+
282
+
```bash
283
+
curl -X POST "http://localost:8000/text-to-image" -H "Content-Type: application/json" -d '{"prompt": "A cool cat on the beach."}'
284
+
```
285
+
286
+
This sends a POST request to the `text-to-image` endpoint on the worker node with the specified JSON payload.
287
+
</Step>
288
+
<Steptitle="Inspect the Response">
289
+
If the AI Worker node is functioning correctly, you should receive a response similar to this:
290
+
291
+
```json
292
+
{
293
+
"images": [
294
+
{
295
+
"url": "data:image/png;base64,iVBORw0KGgoAA...",
296
+
"seed": 2724904334
297
+
}
298
+
]
299
+
}
300
+
```
301
+
302
+
As with the Swagger UI response, the `url` field contains a base64 encoded image that can be decoded into PNG format using a tool like [Base64.guru](https://base64.guru/converter/decode/image/png).
0 commit comments