Skip to content

Commit 938abb2

Browse files
rickstaarickstaa
authored
docs(ai): add upscaling pipeline (#585)
* docs(ai): add upscaling pipeline This commit adds the documentation for the new upscaling pipeline to the docs. * docs(ai): add nsfw to pipeline response This commit adds the new nsfw parameter in the response displayed in the pipeline documentation. * docs(ai): fix upscale diagram image path This commit ensures that the right image path is used in the mermaid graph.
1 parent 8a72424 commit 938abb2

9 files changed

+216
-6
lines changed

‎ai/api-reference/ai-openapi-schema.yml

+63
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,43 @@ paths:
118118
$ref: '#/components/schemas/HTTPValidationError'
119119
security:
120120
- HTTPBearer: []
121+
/upscale:
122+
post:
123+
summary: Upscale
124+
operationId: upscale
125+
requestBody:
126+
content:
127+
multipart/form-data:
128+
schema:
129+
$ref: '#/components/schemas/Body_upscale_upscale_post'
130+
required: true
131+
responses:
132+
'200':
133+
description: Successful Response
134+
content:
135+
application/json:
136+
schema:
137+
$ref: '#/components/schemas/ImageResponse'
138+
'400':
139+
description: Bad Request
140+
content:
141+
application/json:
142+
schema:
143+
$ref: '#/components/schemas/HTTPError'
144+
'500':
145+
description: Internal Server Error
146+
content:
147+
application/json:
148+
schema:
149+
$ref: '#/components/schemas/HTTPError'
150+
'422':
151+
description: Validation Error
152+
content:
153+
application/json:
154+
schema:
155+
$ref: '#/components/schemas/HTTPValidationError'
156+
security:
157+
- HTTPBearer: []
121158
components:
122159
schemas:
123160
APIError:
@@ -217,6 +254,32 @@ components:
217254
- image
218255
- model_id
219256
title: Body_image_to_video_image_to_video_post
257+
Body_upscale_upscale_post:
258+
properties:
259+
prompt:
260+
type: string
261+
title: Prompt
262+
image:
263+
type: string
264+
format: binary
265+
title: Image
266+
model_id:
267+
type: string
268+
title: Model Id
269+
default: ''
270+
safety_check:
271+
type: boolean
272+
title: Safety Check
273+
default: true
274+
seed:
275+
type: integer
276+
title: Seed
277+
type: object
278+
required:
279+
- prompt
280+
- image
281+
- model_id
282+
title: Body_upscale_upscale_post
220283
HTTPError:
221284
properties:
222285
detail:

‎ai/api-reference/upscale.mdx

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
openapi: post /upscale
3+
---
4+
5+
<Info>
6+
The public [Livepeer.cloud](https://www.livepeer.cloud/) Gateway used in this
7+
guide is intended for experimentation and is not guaranteed for production
8+
use. It is a free, non-token-gated, but rate-limited service designed for
9+
testing purposes. For production-ready applications, consider setting up your
10+
own Gateway node or partnering with one via the `ai-video` channel on
11+
[Discord](https://discord.gg/livepeer).
12+
</Info>
13+
14+
<Note>
15+
Please note that the **optimal** parameters for a given model may vary
16+
depending on the specific model and use case. The parameters provided in this
17+
guide are not model-specific and should be used as a starting point.
18+
Additionally, some models may have parameters such as `guiding_scale` and
19+
`num_inference_steps` disabled by default. For more information on
20+
model-specific parameters, please refer to the respective model documentation.
21+
</Note>

‎ai/introduction.mdx

+6-5
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,12 @@ The **AI Video Subnet**, also known as the **AI Subnet**, is the first step
1616
toward bringing powerful AI video capabilities into the Livepeer network. It
1717
enables video developers to add a rapidly growing suite of **generative AI
1818
features** such as [text-to-image](/ai/pipelines/text-to-image),
19-
[image-to-image](/ai/pipelines/image-to-image) and
20-
[image-to-video](/ai/pipelines/image-to-video) to their applications. Livepeer
21-
Node operators are able to **earn revenue by deploying their GPU resources** for
22-
AI processing tasks. Ready to dive in? Choose one of the cards below to
23-
kickstart your journey with the AI Subnet.
19+
[image-to-image](/ai/pipelines/image-to-image),
20+
[image-to-video](/ai/pipelines/image-to-video) and
21+
[upscaling](/ai/pipelines/upscale) to their applications. Livepeer Node
22+
operators are able to **earn revenue by deploying their GPU resources** for AI
23+
processing tasks. Ready to dive in? Choose one of the cards below to kickstart
24+
your journey with the AI Subnet.
2425

2526
## Kickstart Your Journey
2627

‎ai/pipelines/image-to-image.mdx

+1
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ to the Gateway:
110110
{
111111
"images": [
112112
{
113+
"nsfw": false,
113114
"seed": 3197613440,
114115
"url": "https://<gateway-ip>/stream/dd5ad78d/7adde483.png"
115116
}

‎ai/pipelines/image-to-video.mdx

+1
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ to the Gateway:
9999
{
100100
"images": [
101101
{
102+
"nsfw": false,
102103
"seed": 1914955328,
103104
"url": "/stream/2b835716/01c0e9a6.mp4"
104105
}

‎ai/pipelines/text-to-image.mdx

+1
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,7 @@ to the Gateway:
117117
{
118118
"images": [
119119
{
120+
"nsfw": false,
120121
"seed": 2562822894,
121122
"url": "https://<gateway-ip>/stream/d0fc1fc6/8fdf5a94.png"
122123
}

‎ai/pipelines/upscale.mdx

+121
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
---
2+
title: Upscale
3+
---
4+
5+
## Overview
6+
7+
The AI Subnet's `upscale` pipeline provides **advanced image upscaling**.
8+
Powered by the latest diffusion models in HuggingFace's
9+
[super-resolution](https://huggingface.co/docs/diffusers/en/api/pipelines/stable_diffusion/upscale)
10+
pipeline, it enhances the resolution of input images by a factor of 4.
11+
12+
<div align="center">
13+
14+
{/* TODO: Replace with relative url when mintlify fixed issue. */}
15+
16+
```mermaid
17+
graph LR
18+
A[<div style="width: 128px;"><img src="https://mintlify.s3-us-west-1.amazonaws.com/na-36-ai-video/images/ai/cool-cat-low-res.png" alt="Image of low resolution cat"/></div>] --> B[Gateway]
19+
P[A white cat wearing sunglasses on the beach] --> B
20+
B --> C[Orchestrator]
21+
C --> B
22+
B --> D[<div style="width: 200px;"><img src="https://mintlify.s3-us-west-1.amazonaws.com/na-36-ai-video/images/ai/cool-cat.png" alt="Image of high resolution cat"/></div>]
23+
```
24+
25+
</div>
26+
27+
## Models
28+
29+
### Warm Models
30+
31+
The current warm model requested for the `upscale` pipeline is:
32+
33+
- [stabilityai/stable-diffusion-x4-upscaler](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler):
34+
A text-guided upscaling diffusion model trained on large LAION images,
35+
offering enhanced resolution and controlled noise addition.
36+
37+
<Tip>
38+
For faster responses with different
39+
[upscale](https://huggingface.co/docs/diffusers/en/api/pipelines/stable_diffusion/upscale)
40+
compatible diffusion models, ask Orchestrators to load it on their GPU via the
41+
`ai-video` channel in [Discord Server](https://discord.gg/livepeer).
42+
</Tip>
43+
44+
### On-Demand Models
45+
46+
The following models have been tested and verified for the `upscale` pipeline:
47+
48+
<Note>
49+
If a specific model you wish to use is not listed, please submit a [feature
50+
request](https://github.com/livepeer/ai-worker/issues/new?assignees=&labels=enhancement%2Cmodel&projects=&template=model_request.yml)
51+
on GitHub to get the model verified and added to the list.
52+
</Note>
53+
54+
{/* prettier-ignore */}
55+
<Accordion title="Tested and Verified Diffusion Models">
56+
- [stabilityai/stable-diffusion-x4-upscaler](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler):
57+
A text-guided upscaling diffusion model trained on large LAION images,
58+
offering enhanced resolution and controlled noise addition.
59+
</Accordion>
60+
61+
## Basic Usage Instructions
62+
63+
<Tip>
64+
For a detailed understanding of the `upscale` endpoint and to experiment with
65+
the API, see the [AI Subnet API Reference](/ai/api-reference/upscale).
66+
</Tip>
67+
68+
To generate an image with the `upscale` pipeline, send a `POST` request to the
69+
Gateway's `upscale` API endpoint:
70+
71+
```bash
72+
curl -X POST https://<gateway-ip>/upscale \
73+
-F model_id="stabilityai/stable-diffusion-x4-upscaler" \
74+
-F image=@<PATH_TO_IMAGE>/low_res_cat.png \
75+
-F prompt="A white cat"
76+
```
77+
78+
In this command:
79+
80+
- `<gateway-ip>` should be replaced with your AI Gateway's IP address.
81+
- `model_id` is the diffusion model for image generation.
82+
- The `image` field holds the **absolute** path to the image file to be
83+
upscaled.
84+
- `prompt` is a descriptive text that provides context about the content of the
85+
image.
86+
87+
For additional optional parameters, refer to the
88+
[AI Subnet API Reference](/ai/api-reference/upscale).
89+
90+
After execution, the Orchestrator processes the request and returns the response
91+
to the Gateway:
92+
93+
```json
94+
{
95+
"images": [
96+
{
97+
"nsfw": false,
98+
"seed": 3197613440,
99+
"url": "https://<gateway-ip>/stream/dd5ad78d/7adde483.png"
100+
}
101+
]
102+
}
103+
```
104+
105+
The `url` in the response is the URL of the generated image. Download the image
106+
with:
107+
108+
```bash
109+
curl -O "https://<STORAGE_ENDPOINT>/stream/dd5ad78d/7adde483.png"
110+
```
111+
112+
## API Reference
113+
114+
<Card
115+
title="API Reference"
116+
icon="rectangle-terminal"
117+
href="/ai/api-reference/upscale"
118+
>
119+
Explore the `upscale` endpoint and experiment with the API in the AI Subnet
120+
API Reference.
121+
</Card>

‎images/ai/cool-cat-low-res.png

36.3 KB
Loading

‎mint.json

+2-1
Original file line numberDiff line numberDiff line change
@@ -520,7 +520,8 @@
520520
"ai/pipelines/overview",
521521
"ai/pipelines/text-to-image",
522522
"ai/pipelines/image-to-image",
523-
"ai/pipelines/image-to-video"
523+
"ai/pipelines/image-to-video",
524+
"ai/pipelines/upscale"
524525
]
525526
},
526527
{

0 commit comments

Comments
 (0)