[BUG] hauler store sync -f some-yaml-file.yaml --log-level debug seems to hang on last image for 10 minutes

[irishgordo]

Environmental Info:

ubuntu@haul-1-26-2026:~$ uname -a
Linux haul-1-26-2026 6.8.0-100-generic #100-Ubuntu SMP PREEMPT_DYNAMIC Tue Jan 13 16:40:06 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@haul-1-26-2026:~$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Hauler Version:

ubuntu@haul-1-26-2026:~$ hauler version
 __    __       ___       __    __   __       _______ .______
|  |  |  |     /   \     |  |  |  | |  |     |   ____||   _  \
|  |__|  |    /  ^  \    |  |  |  | |  |     |  |__   |  |_)  |
|   __   |   /  /_\  \   |  |  |  | |  |     |   __|  |      /
|  |  |  |  /  _____  \  |  `--'  | |  `----.|  |____ |  |\  \----.
|__|  |__| /__/     \__\  \______/  |_______||_______|| _| `._____|
hauler: Airgap Swiss Army Knife

GitVersion:    v1.4.1

Describe the Bug:

Hey there,
Hauler has been awesome to leverage for us.
We've not been able to just yet port over to using hauler-helm with success but ideally that will be a future state approach for automation that's more solid.

But currently it seems at time that the hauler store sync -f some-yaml-file.yaml --log-level debug hangs for a very long time.
This makes automation hard as we're just waiting for it to return with an exit code 0 but legacy ansible timing and things this is pretty difficult to juggle (again, hauler-helm will be the better solution long term for us 😅 )

It's like:

• on the first run, last image it hung for 3 minutes before interrupt on one run.
• on the second run, last image hung for about 4 minutes before interrupt
• on the third run, wrapping the command in time * (ie: time hauler store sync -f k3s-1-34-5-k3s1.yaml --log-level debug) it hung for NN Minutes

Steps to Reproduce:
(note this might be a bit difficult to reproduce)

1. ensure you avoid dreaded rate-limiting, do a hauler login docker.io -u your-awesome-username
2. previously run hauler store sync -f commands with other manifests , combining like some with others possibly, make sure the manifests are all from docker.io
3. then run a new hauler store sync -f command with just the k3s yaml for the images, wrap it in a time call
4. observe the behavior

Expected Behavior:

• I'm really not sure, it looks like on the debug logs that the call to snag the last image was like done already... or it feels like it should be done? I don't know, pretty greenhorn on the internals here
• I guess, I'm just under the assumption (which yeah assumptions are bad here 😆 ) - that it wouldn't take ~10 minutes on a list of images that are only like 8 deep... and none of those images "seem" to be ultra massive... and we've also done a docker login ... sooooo rate-limiting shouldn't happen

...Or!
Maybe the --log-level debug could be way more verbose to tell us "hey i'm still chomping through this, don't worry 😄 "

Again... yeah hauler-helm is the tool we need to use... as it just simply doesn't make sense for us to stand up and tear down a VM just to slug through needing to acquire 150G of images for all the stuff we gotta airgap. We've had some issues getting hauler-helm (harvester-csi-backed storage on RKE2 on Harvester with HelmChart CRD or some other variety) to stand up correctly with some limited cycles we've had in the midst of release chaos, but not to much avail just yet - hence just the fallback on the VM methodology with the binary itself.

Actual Behavior:

ubuntu@haul-1-26-2026:~$ time hauler store sync -f k3s-1-34-5-k3s1.yaml --log-level debug
2026-04-17 19:38:55 DBG running cli command [hauler store sync]
2026-04-17 19:38:55 DBG using store at [/home/ubuntu/store]
2026-04-17 19:38:55 DBG using temporary directory at [/tmp/hauler2072479124]
2026-04-17 19:38:55 INF processing manifest [k3s-1-34-5-k3s1.yaml] to store [/home/ubuntu/store]
2026-04-17 19:38:55 INF syncing content [content.hauler.cattle.io/v1] with [kind=Images] to store [/home/ubuntu/store]
2026-04-17 19:38:55 INF adding image [index.docker.io/rancher/klipper-helm:v0.9.14-build20260210] to the store
2026-04-17 19:38:56 DBG multi-arch image [true]
2026-04-17 19:38:56 DBG platform for image [linux/amd64]
2026-04-17 19:38:57 INF successfully added image [index.docker.io/rancher/klipper-helm:v0.9.14-build20260210]
2026-04-17 19:38:57 INF adding image [index.docker.io/rancher/klipper-lb:v0.4.14] to the store
2026-04-17 19:38:57 DBG multi-arch image [true]
2026-04-17 19:38:57 DBG platform for image [linux/amd64]
2026-04-17 19:38:59 INF successfully added image [index.docker.io/rancher/klipper-lb:v0.4.14]
2026-04-17 19:38:59 INF adding image [index.docker.io/rancher/local-path-provisioner:v0.0.34] to the store
2026-04-17 19:38:59 DBG multi-arch image [true]
2026-04-17 19:38:59 DBG platform for image [linux/amd64]
2026-04-17 19:39:00 INF successfully added image [index.docker.io/rancher/local-path-provisioner:v0.0.34]
2026-04-17 19:39:00 INF adding image [index.docker.io/rancher/mirrored-coredns-coredns:1.14.1] to the store
2026-04-17 19:39:01 DBG multi-arch image [true]
2026-04-17 19:39:01 DBG platform for image [linux/amd64]
2026-04-17 19:39:05 INF successfully added image [index.docker.io/rancher/mirrored-coredns-coredns:1.14.1]
2026-04-17 19:39:05 INF adding image [index.docker.io/rancher/mirrored-library-busybox:1.37.0] to the store
2026-04-17 19:39:05 DBG multi-arch image [true]
2026-04-17 19:39:05 DBG platform for image [linux/amd64]
2026-04-17 19:39:06 INF successfully added image [index.docker.io/rancher/mirrored-library-busybox:1.37.0]
2026-04-17 19:39:06 INF adding image [index.docker.io/rancher/mirrored-library-traefik:3.6.9] to the store
2026-04-17 19:39:07 DBG multi-arch image [true]
2026-04-17 19:39:07 DBG platform for image [linux/amd64]
2026-04-17 19:39:08 INF successfully added image [index.docker.io/rancher/mirrored-library-traefik:3.6.9]
2026-04-17 19:39:08 INF adding image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1] to the store
2026-04-17 19:39:08 DBG multi-arch image [true]
2026-04-17 19:39:08 DBG platform for image [linux/amd64]
2026-04-17 19:39:09 INF successfully added image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1]
2026-04-17 19:39:09 INF adding image [index.docker.io/rancher/mirrored-pause:3.6] to the store
2026-04-17 19:39:10 DBG multi-arch image [true]
2026-04-17 19:39:10 DBG platform for image [linux/amd64]
2026-04-17 19:39:11 INF successfully added image [index.docker.io/rancher/mirrored-pause:3.6]
2026-04-17 19:48:39 INF processing completed successfully

real    9m43.402s
user    9m52.898s
sys     1m46.067s
ubuntu@haul-1-26-2026:~$ 

Additional Context:

• inside a tmux session

2026-04-17 19:30:12 INF adding image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1] to the store
2026-04-17 19:30:12 DBG multi-arch image [true]
2026-04-17 19:30:12 DBG platform for image [linux/amd64]
2026-04-17 19:30:13 INF successfully added image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1]
2026-04-17 19:30:13 INF adding image [index.docker.io/rancher/mirrored-pause:3.6] to the store
2026-04-17 19:30:14 DBG multi-arch image [true]
2026-04-17 19:30:14 DBG platform for image [linux/amd64]
2026-04-17 19:30:15 INF successfully added image [index.docker.io/rancher/mirrored-pause:3.6]





^C
ubuntu@haul-1-26-2026:~$ hauler store sync -f k3s-1-34-5-k3s1.yaml --log-level debug
2026-04-17 19:33:56 DBG running cli command [hauler store sync]
2026-04-17 19:33:56 DBG using store at [/home/ubuntu/store]
2026-04-17 19:33:56 DBG using temporary directory at [/tmp/hauler338158020]
2026-04-17 19:33:56 INF processing manifest [k3s-1-34-5-k3s1.yaml] to store [/home/ubuntu/store]
2026-04-17 19:33:56 INF syncing content [content.hauler.cattle.io/v1] with [kind=Images] to store [/home/ubuntu/store]
2026-04-17 19:33:56 INF adding image [index.docker.io/rancher/klipper-helm:v0.9.14-build20260210] to the store
2026-04-17 19:33:56 DBG multi-arch image [true]
2026-04-17 19:33:56 DBG platform for image [linux/amd64]
2026-04-17 19:33:58 INF successfully added image [index.docker.io/rancher/klipper-helm:v0.9.14-build20260210]
2026-04-17 19:33:58 INF adding image [index.docker.io/rancher/klipper-lb:v0.4.14] to the store
2026-04-17 19:33:58 DBG multi-arch image [true]
2026-04-17 19:33:58 DBG platform for image [linux/amd64]
2026-04-17 19:33:59 INF successfully added image [index.docker.io/rancher/klipper-lb:v0.4.14]
2026-04-17 19:33:59 INF adding image [index.docker.io/rancher/local-path-provisioner:v0.0.34] to the store
2026-04-17 19:33:59 DBG multi-arch image [true]
2026-04-17 19:33:59 DBG platform for image [linux/amd64]
2026-04-17 19:34:01 INF successfully added image [index.docker.io/rancher/local-path-provisioner:v0.0.34]
2026-04-17 19:34:01 INF adding image [index.docker.io/rancher/mirrored-coredns-coredns:1.14.1] to the store
2026-04-17 19:34:01 DBG multi-arch image [true]
2026-04-17 19:34:01 DBG platform for image [linux/amd64]
2026-04-17 19:34:02 INF successfully added image [index.docker.io/rancher/mirrored-coredns-coredns:1.14.1]
2026-04-17 19:34:02 INF adding image [index.docker.io/rancher/mirrored-library-busybox:1.37.0] to the store
2026-04-17 19:34:03 DBG multi-arch image [true]
2026-04-17 19:34:03 DBG platform for image [linux/amd64]
2026-04-17 19:34:04 INF successfully added image [index.docker.io/rancher/mirrored-library-busybox:1.37.0]
2026-04-17 19:34:04 INF adding image [index.docker.io/rancher/mirrored-library-traefik:3.6.9] to the store
2026-04-17 19:34:04 DBG multi-arch image [true]
2026-04-17 19:34:04 DBG platform for image [linux/amd64]
2026-04-17 19:34:05 INF successfully added image [index.docker.io/rancher/mirrored-library-traefik:3.6.9]
2026-04-17 19:34:05 INF adding image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1] to the store
2026-04-17 19:34:06 DBG multi-arch image [true]
2026-04-17 19:34:06 DBG platform for image [linux/amd64]
2026-04-17 19:34:07 INF successfully added image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1]
2026-04-17 19:34:07 INF adding image [index.docker.io/rancher/mirrored-pause:3.6] to the store
2026-04-17 19:34:07 DBG multi-arch image [true]
2026-04-17 19:34:07 DBG platform for image [linux/amd64]
2026-04-17 19:34:09 INF successfully added image [index.docker.io/rancher/mirrored-pause:3.6]




^C
ubuntu@haul-1-26-2026:~$ time hauler store sync -f k3s-1-34-5-k3s1.yaml --log-level debug
2026-04-17 19:38:55 DBG running cli command [hauler store sync]
2026-04-17 19:38:55 DBG using store at [/home/ubuntu/store]
2026-04-17 19:38:55 DBG using temporary directory at [/tmp/hauler2072479124]
2026-04-17 19:38:55 INF processing manifest [k3s-1-34-5-k3s1.yaml] to store [/home/ubuntu/store]
2026-04-17 19:38:55 INF syncing content [content.hauler.cattle.io/v1] with [kind=Images] to store [/home/ubuntu/store]
2026-04-17 19:38:55 INF adding image [index.docker.io/rancher/klipper-helm:v0.9.14-build20260210] to the store
2026-04-17 19:38:56 DBG multi-arch image [true]
2026-04-17 19:38:56 DBG platform for image [linux/amd64]
2026-04-17 19:38:57 INF successfully added image [index.docker.io/rancher/klipper-helm:v0.9.14-build20260210]
2026-04-17 19:38:57 INF adding image [index.docker.io/rancher/klipper-lb:v0.4.14] to the store
2026-04-17 19:38:57 DBG multi-arch image [true]
2026-04-17 19:38:57 DBG platform for image [linux/amd64]
2026-04-17 19:38:59 INF successfully added image [index.docker.io/rancher/klipper-lb:v0.4.14]
2026-04-17 19:38:59 INF adding image [index.docker.io/rancher/local-path-provisioner:v0.0.34] to the store
2026-04-17 19:38:59 DBG multi-arch image [true]
2026-04-17 19:38:59 DBG platform for image [linux/amd64]
2026-04-17 19:39:00 INF successfully added image [index.docker.io/rancher/local-path-provisioner:v0.0.34]
2026-04-17 19:39:00 INF adding image [index.docker.io/rancher/mirrored-coredns-coredns:1.14.1] to the store
2026-04-17 19:39:01 DBG multi-arch image [true]
2026-04-17 19:39:01 DBG platform for image [linux/amd64]
2026-04-17 19:39:05 INF successfully added image [index.docker.io/rancher/mirrored-coredns-coredns:1.14.1]
2026-04-17 19:39:05 INF adding image [index.docker.io/rancher/mirrored-library-busybox:1.37.0] to the store
2026-04-17 19:39:05 DBG multi-arch image [true]
2026-04-17 19:39:05 DBG platform for image [linux/amd64]
2026-04-17 19:39:06 INF successfully added image [index.docker.io/rancher/mirrored-library-busybox:1.37.0]
2026-04-17 19:39:06 INF adding image [index.docker.io/rancher/mirrored-library-traefik:3.6.9] to the store
2026-04-17 19:39:07 DBG multi-arch image [true]
2026-04-17 19:39:07 DBG platform for image [linux/amd64]
2026-04-17 19:39:08 INF successfully added image [index.docker.io/rancher/mirrored-library-traefik:3.6.9]
2026-04-17 19:39:08 INF adding image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1] to the store
2026-04-17 19:39:08 DBG multi-arch image [true]
2026-04-17 19:39:08 DBG platform for image [linux/amd64]
2026-04-17 19:39:09 INF successfully added image [index.docker.io/rancher/mirrored-metrics-server:v0.8.1]
2026-04-17 19:39:09 INF adding image [index.docker.io/rancher/mirrored-pause:3.6] to the store
2026-04-17 19:39:10 DBG multi-arch image [true]
2026-04-17 19:39:10 DBG platform for image [linux/amd64]
2026-04-17 19:39:11 INF successfully added image [index.docker.io/rancher/mirrored-pause:3.6]

Here's the manifest, the k3s blob of images that we're trying to airgap

apiVersion: content.hauler.cattle.io/v1
kind: Images
metadata:
  annotations:
    hauler.dev/platform: linux/amd64
    hauler.dev/registry: docker.io
  name: k3s-1-34-5-k3s1-images
spec:
  images:
    - name: rancher/klipper-helm:v0.9.14-build20260210
    - name: rancher/klipper-lb:v0.4.14
    - name: rancher/local-path-provisioner:v0.0.34
    - name: rancher/mirrored-coredns-coredns:1.14.1
    - name: rancher/mirrored-library-busybox:1.37.0
    - name: rancher/mirrored-library-traefik:3.6.9
    - name: rancher/mirrored-metrics-server:v0.8.1
    - name: rancher/mirrored-pause:3.6

Y'all are awesome!

• we really appreciate all the work that's gone into this project and it's been a crucial tool for us

[irishgordo]

cc: @khushboo-rancher @TachunLin @asc5543 @noahgildersleeve @albinsun @lanfon72 @wheatdog

[amartin120]

@irishgordo Hi there. I have a feeling that I know what's happening here and we may have already solved this with our upcoming v2 Hauler. I'd be curious if you would be willing to test out our v2.0.0-dev.2 dev release to see if it resolves your problem.

v2 has an overhaul that includes a change to what happens at the end of a sync that we weren't really able to work around consistently with v1.