Docker Hub coupling makes the UI unusable for Fusion-only / air-gapped deployments

[antoniocali]

Summary

The UI server hard-couples several code paths to hub.docker.com (the metadata API, not the registry), which is fragile in any deployment where outbound to Docker Hub is rate-limited, slow, or restricted — and is dead weight in Fusion-only deployments that never run CDC sync jobs.

I'm running OLake on Kubernetes purely as the Iceberg maintenance layer (Fusion + Spark optimizer) against an existing Polaris catalog. No olakeWorker CDC, no source / destination connectors. Observed three classes of issue:

1. Continuous background polling generates log noise

Every ~30s the UI polls Docker Hub for tags on every CDC source connector image, even when CDC is unused:

2026-05-21T15:41:22Z WRN failed to fetch image tags online for olakego/source-mysql:
  docker hub api request failed with status code: 504.
  Cached fallback unavailable on Kubernetes (no Docker daemon)
2026-05-21T15:41:52Z WRN failed to fetch image tags online for olakego/source-postgres: …
2026-05-21T15:42:22Z WRN failed to fetch image tags online for olakego/source-oracle: …
2026-05-21T15:42:52Z WRN failed to fetch image tags online for olakego/source-mongodb: …
… (also kafka, s3, db2, mssql)

Eight connectors × ~30s polling = ~16 calls/min hitting Docker Hub. With the (very strict) anonymous metadata-API rate limits, these mostly 504 and just generate continuous warnings.

The Cached fallback unavailable on Kubernetes (no Docker daemon) text is also confusing — it implies a feature is missing on K8s rather than that the deployment doesn't need it.

2. User-facing endpoints become slow / fail when Docker Hub throttles

When the rate limit is in effect, user-facing endpoints that also call Docker Hub become very slow or 500:

[GIN] 2026/05/21 - 15:36:48 | 500 |   4m0s    | GET  /api/v1/project/123/destinations/versions?type=iceberg
[GIN] 2026/05/21 - 15:38:55 | 500 |   30.1s   | GET  /api/v1/project/123/sources/versions?type=mongodb
[GIN] 2026/05/21 - 15:40:49 | 500 |   4m0s    | GET  /api/v1/project/123/destinations/versions?type=iceberg
[GIN] 2026/05/21 - 20:58:23 | 200 |  15.58s   | POST /api/v1/project/123/destinations/spec

POST /destinations/spec (server/internal/services/etl/destination.go:264) calls utils.GetDriverImageTags(...) before doing the actual work — even though the existing // TODO: cache spec in db for each version (line 263) acknowledges this.

The four-minute hangs are particularly bad — the user clicks a tab and the UI just spins until they assume something's broken.

3. No way to disable connector discovery

There's no config flag to disable Docker Hub polling, and useStandardResources: true / K8s mode doesn't gate it. Deployments that only use Fusion (or pin specific connector versions, or run air-gapped against a private registry mirror) get no benefit from the polling but pay the full cost.

Suggested fixes (in order of value)

1. Add a connectors.discoveryEnabled (or similar) chart/server config flag that disables the background polling and the eager Docker Hub calls in /spec, /versions, and /releases. Default true for backwards compat; set to false for Fusion-only deployments. Probably worth defaulting false when running on K8s where the "Docker daemon cached fallback" path doesn't apply anyway.

2. Back off on 504s instead of polling every 30s. Exponential backoff after a few consecutive failures would reduce log noise dramatically without any feature change.

3. Cache driver specs in postgres (the existing TODO at destination.go:263). One-shot fetch on first use; subsequent calls served from DB. This makes the user-facing slowness a one-time cost per version and survives Docker Hub being down.

4. Allow pointing connector discovery at a custom registry/mirror. Useful for air-gapped or rate-limit-managed deployments — same shape as image.repository overrides but for the discovery API. (Helm chart side; would need server changes too.)

5. Reword the K8s warning. Cached fallback unavailable on Kubernetes (no Docker daemon) is technically accurate but reads like a deficiency. Something like Docker Hub query failed; no local cache available in K8s mode — connector discovery unavailable, this is expected if connectors are unused would be less alarming.

I'm happy to send a PR for (1) — gate the polling and the eager /spec and /versions Docker-Hub calls behind a config flag — if there's interest.

Environment

• olake/olake Helm chart 0.0.18 on EKS
• Fusion enabled (fusion.enabled: true), no CDC source/destination configured
• Polaris (Snowflake-hosted) as the Iceberg REST catalog
• Outbound to registry-1.docker.io (image pulls) works; outbound to hub.docker.com (metadata API) intermittently 504s

[vishalm0509]

@antoniocali
Regarding the issue:

1. Continuous background polling generates log noise
   Every ~30s the UI polls Docker Hub for tags on every CDC source connector image, even when CDC is unused:

For catalog creation /api/v1/project/123/destination/spec, Olake currently requires a source driver image to initiate the operation (to get the JSONSchema for the form). At the moment, the system first tries the MySQL image as the default and, if not available locally, checks other driver images one by one.

Based on the logs you shared, Docker Hub is returning 503 errors. As a result, the MySQL image pull fails, after which the system attempts the remaining driver images sequentially, eventually failing for all of them.

---

The Cached fallback unavailable on Kubernetes (no Docker daemon) text is also confusing — it implies a feature is missing on K8s rather than that the deployment doesn't need it.

In offline environments (air-gapped) we first try online fetch, and if failed we check the images present locally. Since in k8s environment local image fetch feature is not yet supported, you get the WARN log.
In case we want to support this you can try using a private container registry:
https://olake.io/docs/install/kubernetes-compaction/#private-container-registry
https://olake.io/docs/install/olake-ui/offline-environments-aws/#3-create-the-ecr-pull-through-cache-rule

---

2. User-facing endpoints become slow / fail when Docker Hub throttles
   When the rate limit is in effect, user-facing endpoints that also call Docker Hub become very slow or 500:

In order to initiate the operations spec - we need the driver image, that's the reason we fetch the image first.

---

3. No way to disable connector discovery

We already have improvements planned in our roadmap to further optimize this flow, including:
separating Fusion test connection from driver images
improving spec lookup performance through caching

---

From the issue description, it looks like the registry you configured is directly forwarding requests to Docker Hub. Could you please confirm whether you are using an offline/air-gapped setup? Since this setup is not fully supported yet, could you try using an ECR-based setup instead?

[antoniocali]

Thanks for the detailed reply!

Let me step back and make my use case clearer, because I think the air-gapped framing is a misread of my setup.

I am running Fusion only. No CDC, no ingestion, no sources, no destinations. The only thing I use OLake for is Iceberg table maintenance (compaction, snapshot expiry, orphan files) against tables that are already created and written to by other systems. So:

• I will never click "Add source" or "Add destination" in the UI.
• I do not need any driver images (mysql, postgres, mongodb, kafka, etc.).
• I do not need the spec form, version dropdowns, or any connector discovery.

I only use the Optimizer / Catalogs side of the UI.

My environment is not air-gapped. The cluster has open egress to the internet. Image pulls from registry-1.docker.io work fine — postgres, temporal, olake-ui, fusion, and the Spark optimizer images all pull cleanly. So the ECR pull-through cache suggestion does not apply to my situation.

The actual problem is different. The UI server (Go) makes outbound HTTP calls to hub.docker.com/v2/repositories/olakego/source-*/tags/ — the Docker Hub metadata API, not the image registry. Docker Hub rate-limits anonymous calls on that API very aggressively, so most of these calls come back 504. An ECR pull-through cache only proxies image pulls (registry endpoints); it does not intercept calls the Go server makes to hub.docker.com. So pointing at ECR would not change anything here.

About the 30s polling: I think this got mixed with the /destinations/spec explanation in your reply, but they are two different code paths. The /spec slowness only happens when a user clicks something in the UI. The bigger noise source is a background loop that runs every ~30 seconds for each of the 8 source drivers, even when I am not clicking anything. That is what fills the logs with failed to fetch image tags online for olakego/source-mysql warnings 24/7. For someone running Fusion only, this is pure waste.

The roadmap items you mentioned are exactly what I am hoping for:

1. Separating Fusion test connection from driver images — yes, please. Fusion does not need driver images at all to test or operate.
2. Improving spec lookup performance through caching — also great, though for me an even better fix would be to not call /spec at all in Fusion-only mode.

If you can point me at the tracking issues / PRs for those, I will follow along.

In the meantime, I already opened #382 which adds a CONNECTOR_DISCOVERY_ENABLED server flag (default true to preserve current behavior). When set to false, it short-circuits GetDriverImageTags and uses a configurable DEFAULT_CONNECTOR_VERSION instead of calling Docker Hub. That kills the 30s polling, the /spec Docker Hub call, and the /versions Docker Hub call for deployments that do not use CDC — without changing anything for existing users. Would love a review on that whenever you have time.

Thanks again for engaging with this!

[vishalm0509]

@antoniocali Thanks for the detailed explanation of the issue and opening a PR. We are looking into the issue closely and will get back to you shortly.

In the meantime, could you let us know whether this issue is currently blocking your usage of OLake Fusion? Were you able to run any compaction jobs successfully so far?

[antoniocali]

@antoniocali Thanks for the detailed explanation of the issue and opening a PR. We are looking into the issue closely and will get back to you shortly.

In the meantime, could you let us know whether this issue is currently blocking your usage of OLake Fusion? Were you able to run any compaction jobs successfully so far?

No it doesn't block me.
I am testing in these days the iceberg management via Fusion.

[arsalan-datazip]

@antoniocali Thanks for the detailed explanation of the issue and opening a PR. We are looking into the issue closely and will get back to you shortly.
In the meantime, could you let us know whether this issue is currently blocking your usage of OLake Fusion? Were you able to run any compaction jobs successfully so far?

No it doesn't block me. I am testing in these days the iceberg management via Fusion.

@antoniocali
We have optimized the flow for fusion test connection and spec lookup now it will not pull the driver image, it will released today, No dependency will be there on ingestion (olake go).

[arsalan-datazip]

Hi @antoniocali,

We have released a fix for the fusion test connection issue. Could you please update both olake-ui and fusion to the latest versions and verify the fix and let us know.

For the upgrade steps, please refer to the documentation: https://olake.io/docs/install/olake-ui/#updating-olake-ui-version