| title |
|---|
Start your AI Orchestrator |
The Livepeer AI network is currently in Beta but is already integrated into the main go-livepeer software. You can run the Livepeer AI software using one of the following methods:
- Docker (Recommended): The simplest and preferred method.
- Pre-built Binaries: An alternative if you prefer not to use Docker.
In the Livepeer AI network, orchestrator operations rely on two primary node types:
- Orchestrator: Manages and routes incoming jobs to available compute resources.
- Worker: Performs the actual computation tasks.
The simplest configuration combines both roles on a single machine, utilizing
the machine's GPUs for AI inference tasks, where the orchestrator also functions
as a worker (known as a combined AI orchestrator). In this setup, capacity
is limited by the available GPUs and is set as
runner container count per pipeline/model_id = capacity per pipeline/model_id.
For expanded scalability, operators can deploy dedicated (remote) worker nodes
that connect to the orchestrator, increasing overall compute capacity.
Instructions for setting up remote workers are available on the
next page.
Please follow the steps below to start your combined AI orchestrator node.
Fetch the latest Livepeer AI Docker image from the [Livepeer Docker Hub](https://hub.docker.com/r/livepeer/go-livepeer) with the following command: ```bash
docker pull livepeer/go-livepeer:master
```
</Step>
<Step title="Fetch the Latest AI Runner Docker Image">
The Livepeer AI network employs a [containerized workflow](https://www.ibm.com/topics/containerization) for running AI models. Fetch the latest [AI Runner](https://hub.docker.com/r/livepeer/ai-runner) image with this command:
```bash
docker pull livepeer/ai-runner:latest
```
</Step>
<Step title="Pull Pipeline-Specific Images (optional)">
Next, pull any pipeline-specific images if needed. Check the [pipelines](/ai/pipelines/overview) documentation for more information. For example, to pull the image for the [segment-anything-2](/ai/pipelines/segment-anything-2#pipeline-specific-image) pipeline:
```bash
docker pull livepeer/ai-runner:segment-anything-2
```
</Step>
<Step title="Verify the AI Models are Available">
The Livepeer AI network leverages pre-trained AI models for inference tasks. Before launching the AI Orchestrator node, verify that the weights of these models are accessible on your machine. For more information, visit the [Download AI Models](/ai/orchestrators/models-download) page.
</Step>
<Step title="Configure your AI Orchestrator">
Confirm that the AI models are correctly set up in the `aiModels.json` file in the `~/.lpData/` directory. For guidance on configuring the `aiModels.json` file, refer to the [AI Models Configuration](/ai/orchestrators/models-config) page. The configuration file should resemble:
```json
[
{
"pipeline": "text-to-image",
"model_id": "ByteDance/SDXL-Lightning",
"price_per_unit": 4768371,
"warm": true
}
]
```
</Step>
<Step title="Launch an (off-chain) AI Orchestrator">
Execute the Livepeer AI Docker image using the following command:
```bash
docker run \
--name livepeer_ai_orchestrator \
-v ~/.lpData/:/root/.lpData/ \
-v /var/run/docker.sock:/var/run/docker.sock \
--network host \
--gpus all \
livepeer/go-livepeer:master \
-orchestrator \
-serviceAddr 0.0.0.0:8936 \
-v 6 \
-nvidia 0 \
-aiWorker \
-aiModels /root/.lpData/aiModels.json \
-aiModelsDir ~/.lpData/models \
-aiRunnerImage livepeer/ai-runner:latest # OPTIONAL
```
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
- `-aiWorker`: This flag enables the AI Worker functionality.
- `-aiModels`: This flag sets the path to the JSON file that contains the AI models.
- `-aiModelsDir`: This flag indicates the directory where the AI models are stored on the host machine.
- `-aiRunnerImage`: This optional flag specifies which version of the ai-runner image is used. Example: `livepeer/ai-runner:0.0.2`
Moreover, the `--network host` flag facilitates communication between the AI Orchestrator and the AI Runner container.
Lastly, the `-nvidia` can be configured in a few ways. Use a comma seperated list of GPUs ie. `0,1` to activate specific GPU slots, each GPU will need it's own config item in `aiModels.json`. Alternativly we can use `"all"` to activate all GPUs on the machine with a single model loaded in `aiModels.json` (Warning: If different RAM size GPUs are installed it may cause containers to fail if they have less than the required RAM).
<Warning>Please note that since we use [docker-out-of-docker](https://tdongsi.github.io/blog/2017/04/23/docker-out-of-docker/), the `aiModelsDir` path should be defined as being on the host machine.</Warning>
</Step>
<Step title="Confirm Successful Startup of the AI Orchestrator">
If your Livepeer AI Orchestrator node is functioning correctly, you should see the following output:
```bash
2024/05/01 09:01:39 INFO Starting managed container gpu=0 name=text-to-image_ByteDance_SDXL-Lightning modelID=ByteDance/SDXL-Lightning
...
I0405 22:03:17.427058 2655655 rpc.go:301] Connecting RPC to uri=https://0.0.0.0:8936
I0405 22:03:17.430371 2655655 rpc.go:254] Received Ping request
```
</Step>
<Step title="Check Port Availability">
To make your Livepeer AI Orchestrator node accessible from the internet, you need to configure your network settings. Ensure that port `8936` is unblocked on your machine. Additionally, consider setting up port forwarding on your router, allowing the Gateway node to be reachable from the internet.
</Step>
</Steps>
</Tab>
<Tab title="Use Binaries">
<Steps>
<Step title="Download the Latest Livepeer AI Binary">
Download the latest Livepeer AI binary for your system:
```bash
wget https://build.livepeer.live/go-livepeer/livepeer-<OS>-gpu-<ARCH>.tar.gz
```
Replace `<OS>` and `<ARCH>` with your system's operating system and architecture. For example, for a Linux system with an AMD64 architecture, the command would be:
```bash
wget https://build.livepeer.live/go-livepeer/livepeer-linux-gpu-amd64.tar.gz
```
See the [go-livepeer installation guide](/orchestrators/guides/install-go-livepeer#install-using-a-binary-release) for more information on the available binaries.
<Info>The Windows and MacOS (amd64) binaries of **Livepeer AI** are not available yet.</Info>
</Step>
<Step title="Extract and Configure the Binary">
Once downloaded, extract the binary to a directory of your choice.
</Step>
<Step title="Fetch the Latest AI Runner Docker Image">
The Livepeer AI network employs a [containerized workflow](https://www.ibm.com/topics/containerization) for running AI models. Fetch the latest [AI Runner](https://hub.docker.com/r/livepeer/ai-runner) image with this command:
```bash
docker pull livepeer/ai-runner:latest
```
</Step>
<Step title="Pull Pipeline-Specific Images (optional)">
Next, pull any pipeline-specific images if needed. Check the [pipelines](/ai/pipelines/overview) documentation for more information. For example, to pull the image for the [segment-anything-2](/ai/pipelines/segment-anything-2#pipeline-specific-image) pipeline:
```bash
docker pull livepeer/ai-runner:segment-anything-2
```
</Step>
<Step title="Verify the AI Models are Available">
The Livepeer AI network leverages pre-trained AI models for inference tasks. Before launching the AI Orchestrator node, verify that the weights of these models are accessible on your machine. For more information, visit the [Download AI Models](/ai/orchestrators/models-download) page.
</Step>
<Step title="Configure your AI Orchestrator">
Confirm that the AI models are correctly set up in the `aiModels.json` file in the `~/.lpData/` directory. For guidance on configuring the `aiModels.json` file, refer to the [AI Models Configuration](/ai/orchestrators/models-config) page. The configuration file should resemble:
```json
[
{
"pipeline": "text-to-image",
"model_id": "ByteDance/SDXL-Lightning",
"price_per_unit": 4768371,
"warm": true
}
]
```
</Step>
<Step title="Launch an (off-chain) AI Orchestrator">
Run the following command to start your Livepeer AI Orchestrator node:
```bash
./livepeer \
-orchestrator \
-serviceAddr 0.0.0.0:8936 \
-v 6 \
-nvidia "all" \
-aiWorker \
-aiModels ~/.lpData/aiModels.json \
-aiModelsDir ~/.lpData/models \
-aiRunnerImage livepeer/ai-runner:latest # OPTIONAL
```
This command launches an **off-chain** AI Orchestrator node. While most of the commands are similar to those used when operating a Mainnet Transcoding Network Orchestrator node (explained in the [go-livepeer CLI reference](/references/go-livepeer/cli-reference)), there are a few **Livepeer AI** specific flags:
- `-aiWorker`: This flag enables the AI Worker functionality.
- `-aiModels`: This flag sets the path to the JSON file that contains the AI models.
- `-aiModelsDir`: This flag indicates the directory where the AI models are stored.
- `-aiRunnerImage`: This optional flag specifies which version of the ai-runner image is used. Example: `livepeer/ai-runner:0.0.2`
</Step>
<Step title="Confirm Successful Startup of the AI Orchestrator">
If your Livepeer AI Orchestrator node is functioning correctly, you should see the following output:
```bash
2024/05/01 09:01:39 INFO Starting managed container gpu=0 name=text-to-image_ByteDance_SDXL-Lightning modelID=ByteDance/SDXL-Lightning
...
I0405 22:03:17.427058 2655655 rpc.go:301] Connecting RPC to uri=https://0.0.0.0:8936
I0405 22:03:17.430371 2655655 rpc.go:254] Received Ping request
```
</Step>
<Step title="Check Port Availability">
To make your Livepeer AI Orchestrator node accessible from the internet, you need to configure your network settings. Ensure that port `8936` is unblocked on your machine. Additionally, consider setting up port forwarding on your router, allowing the Gateway node to be reachable from the internet.
</Step>
</Steps>
<Note>
If no binaries are available for your system, you can build the [master branch](https://github.com/livepeer/go-livepeer/tree/master) of [go-livepeer](https://github.com/livepeer/go-livepeer) from source by following the instructions in the [Livepeer repository](/orchestrators/guides/install-go-livepeer) or by reaching out to the Livepeer community on [Discord](https://discord.gg/livepeer).
</Note>
</Tab>
Once your combined Livepeer AI Orchestrator node is running, verify that the
worker is operational by sending an AI inference request directly to the
ai-runner container. You can
either use the Swagger UI interface or
a curl command for this check.
```json
{
"prompt": "A cool cat on the beach."
}
```
This request will instruct the AI model to generate an image based on the text in the `prompt` field.
</Step>
<Step title="Inspect the Inference Response">
If the AI Orchestrator node is functioning correctly, you should receive a response similar to the following:
```json
{
"images": [
{
"url": "data:image/png;base64,iVBORw0KGgoAA...",
"seed": 2724904334
}
]
}
```
The `url` field contains the base64 encoded image generated by the AI model. To convert this image to PNG, use a base64 decoder such as [Base64.guru](https://base64.guru/converter/decode/image/png).
</Step>
</Steps>
</Tab>
<Tab title="Use curl Command">
<Steps>
<Step title="Send an Inference Request with curl">
Alternatively, you can use the `curl` command to test the AI inference capabilities directly. Run the following command, replacing `<WORKER_NODE_IP>` with the IP address of your worker node:
```bash
curl -X POST "http://localost:8000/text-to-image" -H "Content-Type: application/json" -d '{"prompt": "A cool cat on the beach."}'
```
This sends a POST request to the `text-to-image` endpoint on the worker node with the specified JSON payload.
</Step>
<Step title="Inspect the Response">
If the AI Worker node is functioning correctly, you should receive a response similar to this:
```json
{
"images": [
{
"url": "data:image/png;base64,iVBORw0KGgoAA...",
"seed": 2724904334
}
]
}
```
As with the Swagger UI response, the `url` field contains a base64 encoded image that can be decoded into PNG format using a tool like [Base64.guru](https://base64.guru/converter/decode/image/png).
</Step>
</Steps>
</Tab>