Using Hera for inference workloads

[NeoLegends]

Hi!

I'm trying to use Hera to set up a workflow where binary (audio) data is fed in from an HTTP API (via FastAPI), and then a series of postprocessing/inference steps are applied to it, ultimately returning the results to the client via another FastAPI call. In each of these steps I'm trying to use a different base image, as each of the steps requires a different piece of (rather complex) software (torch/Kaldi), that is best managed if left "alone" and isolated in separate container images.

I don't have experience with neither Argo Workflows, nor Hera yet, but it seems like those would be a good fit for this use case, as each processing step will take a couple of minutes to finish.

Having experimented with Hera now for a couple of hours, it seems like I'm hitting a couple of road blocks. Bear with me, these are just my current findings and beliefs about how I understand Hera works, and I hope to be proven wrong here:

1. As far as I've seen the runner setup is the recommended way to go. However, this would require each one of my base images to contain my entire piece of API, which accepts and orchestrates the workflows. This means all the containers require additional dependencies that don't actually matter to them at all. Hera seems to kind of bleed into these containers, otherwise it can't run unless I go back to raw script execution. Is that correct? It would be great to have more docs on how the runner system is implemented under the hood, and what the requirements on my container images are (is installing the hera package enough?) so that the workflows can run in them.
2. Maybe I'm misunderstanding Hera, but it seems rather cumbersome to pass actual binary data between workflow steps. I've seen Custom serialization for non-user types and non-serializable types for Hera runner (Parameter/Artifact inputs and outputs) #1166 (comment). Is using an "input" Annotated[Path, Artifact(output=True)] the recommended way of declaring binary output files these days? The documentation goes into great lengths about passing structured data between workflow steps (it's great that this is so well-supported!), but when it comes to the "simplest" case, raw binary, the docs are surprisingly brief.
3. What is the recommended way of passing artifacts as input to a workflow? Do I upload the data to some S3/MinIO bucket, and then declare an S3Artifact as parameter? The docs are again pretty short on this use case. They mostly go over workflow steps producing artifacts and how those are passed to downstream steps, but what do you do if you want to consume a large binary in your first workflow step? Or do I side-step Argo Workflows entirely on this one, and leave it to the step to get to the binary data via some side-channel (e.g. by passing an HTTP link the step can then fetch into the workflow as parameter)?

Thanks!

[elliotgunton]

Hey @NeoLegends, thanks for trying out Hera! I'll do my best to answer:

1. "this would require each one of my base images to contain my entire piece of API, which accepts and orchestrates the workflows" - yes, the current best practice approach is to contain your business logic and workflow logic under one roof to make building and running it easier. The workflow code is not directly used when the workflow itself is running (i.e. the workflow/DAG is just a definition to pass to Argo Workflows).
   "Hera seems to kind of bleed into these containers, otherwise it can't run unless I go back to raw script execution. Is that correct? " - yes, and assuming you've already read the Runner Script Constructor Guide, there's a bit more about "under the hood" in the API docs. Admittedly these are a bit light on details and it was my intention to provide a full guide to Runner setups (i.e. how/whys of building images, CI/CD, versioning/releasing etc). This example project might help in the meantime!
2. Yes, binary data has been a latecomer to the project because Argo Workflows deals in YAMLs and the conversion between strings/binary can go wrong, so yes, Artifacts are the only way afaik. See the example for loadb and dumpb usage. This is the docs section for passing Artifacts around. (Again, this is a bit light on details, sorry!)
3. "What is the recommended way of passing artifacts as input to a workflow?" - this would always be a case of "my data already exists somewhere (on S3), here's the bucket/key". It's quite cumbersome to set up an existing artifact for examples to work out of the box, so we don't have these examples (yet). "leave it to the step to get to the binary data" - is also a fair approach, depending on how testable you want the code to be and how you want to structure the DAG - e.g. you could construct the required bucket/key strings to pass into the next step, which could then download the artifact itself.

Hope that helps!

[NeoLegends]

Great! Thank you so much for answering with so much detail!

Yes, binary data has been a latecomer to the project because Argo Workflows deals in YAMLs and the conversion between strings/binary can go wrong

If I understand your reply correct you are suggesting I use some sort of "identity" dumpb function that simply takes the file path and reads it from disk into a bytes that Hera will then pick up (and write somewhere else hehe) as artefact?

Currently, when I try to

@script(
    constructor="runner",
    image=API_CONTAINER_IMAGE,
    resources=Resources(cpu_request="250m", memory_request="256Mi"),
)
def fetch_from_url(
    audio_url: str,
    to_file: Annotated[
        Path, Artifact(archive=NoneArchiveStrategy(), output=True)
    ],
):
    import requests

    print(f"loading {audio_url} -> {to_file}")
    r = requests.get(audio_url, stream=True)
    r.raise_for_status()
    with open(to_file, "wb") as out_file:
        for chunk in r.iter_content(chunk_size=256):
            out_file.write(chunk)

I get:

❯ kubectl logs -n argo ableset-lyrixmatch-8kz5w-fetch-from-url-3190421086
time=2026-02-14T10:46:24.954Z level=INFO msg="capturing logs" argo=true
time=2026-02-14T10:46:24.955Z level=INFO msg="waiting for signals" argo=true signalPath=/var/run/argo/ctr/main/signal
loading http://argo-artefacts-minio.default.svc.cluster.local:9000/ableset/in-audio/5c638613-c8f7-48be-b4dd-f866be2b21c4-arabella.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=root%2F20260214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260214T104622Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c34f7fd8b463afd3cf3fb205b662c6011343c4b55b73ff68249466fe782260aa -> /tmp/hera-outputs
time=2026-02-14T10:46:25.955Z level=INFO msg="sub-process exited" argo=true error=<nil>
time=2026-02-14T10:46:25.955Z level=INFO msg="file signal handler exiting due to context cancellation" argo=true
time=2026-02-14T10:46:25.956Z level=INFO msg="saving artifact" dst=/var/run/argo/outputs/artifacts/tmp/hera-outputs/artifacts/to_file.tgz argo=true src=/tmp/hera-outputs/artifacts/to_file
time=2026-02-14T10:46:25.956Z level=INFO msg=taring argo=true source=/tmp/hera-outputs/artifacts/to_file
Error: failed to tarball the output /tmp/hera-outputs/artifacts/to_file to /var/run/argo/outputs/artifacts/tmp/hera-outputs/artifacts/to_file.tgz: stat /tmp/hera-outputs/artifacts/to_file: not a directory
failed to tarball the output /tmp/hera-outputs/artifacts/to_file to /var/run/argo/outputs/artifacts/tmp/hera-outputs/artifacts/to_file.tgz: stat /tmp/hera-outputs/artifacts/to_file: not a directory

Is it a bug that the path's value is /tmp/hera-outputs and not pointing to a file?

EDIT: Yes, I believe this is a bug, it goes away if you give the artifact an explicit name (and stop relying on the parameter's name).

This works 🎉 :

@script(...)
def fetch_from_url(
    ...,
    to_file: Annotated[
        Path, Artifact(archive=NoneArchiveStrategy(), name="to_file", output=True)
    ],
):
    pass

[elliotgunton]

Good to hear it works!

is it a bug that the path's value is /tmp/hera-outputs and not pointing to a file?

Yeah let me confirm that and create an issue