latent-to-image

[hlky]

Context: https://huggingface.slack.com/archives/C065E480NN9/p1738859836468269 (internal)

[apolinario]

Not sure about the new tag, as it maps to the existing tasks and I think we probably don't want to create this as an official task while this is experimental. But maybe this is harmless to exist only in the inference API

[hlky]

Was added for

api-inference-community/api_inference_community/routes.py

Lines 229 to 241 in 329d9bb

	elif task in IMAGE_OUTPUTS:
	image = outputs
	image_format = parse_accept(accept, IMAGE)
	buffer = io.BytesIO()
	image.save(buffer, format=image_format.upper())
	buffer.seek(0)
	img_bytes = buffer.read()
	return Response(
	img_bytes,
	headers=headers,
	status_code=200,
	media_type=f"image/{image_format}",
	)

Seems to be the only usage of IMAGE_OUTPUTS

We also need the task in KNOWN_TASKS (via TENSOR_INPUTS)

api-inference-community/api_inference_community/routes.py

Lines 96 to 102 in 329d9bb

	if task not in KNOWN_TASKS:
	msg = f"The task `{task}` is not recognized by api-inference-community"
	logger.error(msg)
	# Special case: despite the fact that the task comes from environment (which could be considered a service
	# config error, thus triggering a 500), this var indirectly comes from the user
	# so we choose to have a 400 here
	return JSONResponse({"error": msg}, status_code=400)

Should be harmless for this to exist but I agree we don't want it to be an official task while this is experimental.

[Narsil]

We can maybe find ways to "override" another task for the time being (without involving another task)

[Narsil]

Probably not a good place to put the tensor shape in the headers. Why not just put it in the payload ?

I know base64 isn't perfect, but I'd rather go multipart upload than do something like this

[Narsil]

Also changing this requires a release of the api-inference-community package which I think will slow the whole process down (because I will be way more picky about what gets actually merged since it's published as a package therefore harder to take down).

Can we start with base64 and change afterwards for more optimal ?

[hlky]

Changed to use base64.

Minimal example

import io
import base64
import requests
import torch
from PIL import Image
from safetensors.torch import _tobytes
inputs = torch.randn([1, 4, 64, 64], generator=torch.Generator().manual_seed(0))
shape = list(inputs.shape)
dtype = str(inputs.dtype).split(".")[-1]
tensor_data = base64.b64encode(_tobytes(inputs, "inputs")).decode("utf-8")
parameters = {"shape": shape, "dtype": dtype}
data = {"inputs": tensor_data, "parameters": parameters}

response = requests.post("http://127.0.0.1:8000/", json=data)
Image.open(io.BytesIO(response.content))

Generation example

Using Lykon/dreamshaper-8 finetune here for generation, can use stable-diffusion-v1-5/stable-diffusion-v1-5 for VAE.

import io
import base64
import requests
import torch
from PIL import Image
from safetensors.torch import _tobytes

from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "Lykon/dreamshaper-8",
    safety_checker=None,
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

prompt = "a cute cat sitting beside the sea beach."
inputs = pipeline(
    prompt=prompt,
    negative_prompt="bad quality, worse quality, degenerate quality",
    output_type="latent",
).images
shape = list(inputs.shape)
dtype = str(inputs.dtype).split(".")[-1]
tensor_data = base64.b64encode(_tobytes(inputs, "inputs")).decode("utf-8")
parameters = {"shape": shape, "dtype": dtype}
data = {"inputs": tensor_data, "parameters": parameters}

response = requests.post("http://127.0.0.1:8000/", json=data)
Image.open(io.BytesIO(response.content))

[Narsil]

We can maybe find ways to "override" another task for the time being (without involving another task)

[Narsil]

We can definitely assume byteorder is little.