Skip to content

fix: enable chunked blob upload for Azure ACR replication#23016

Open
xrkolovos wants to merge 1 commit intogoharbor:mainfrom
xrkolovos:fix/azurecr-chunked-upload
Open

fix: enable chunked blob upload for Azure ACR replication#23016
xrkolovos wants to merge 1 commit intogoharbor:mainfrom
xrkolovos:fix/azurecr-chunked-upload

Conversation

@xrkolovos
Copy link

Summary

Fixes #23015 — Azure ACR replication fails with HTTP 413 for blob layers larger than ~20MB.
Related: #21640

Root cause

Three issues were identified:

  1. Missing adapter capability declaration — The Azure ACR adapter's Info() method did not set SupportedCopyByChunk: true, so the replication flow always defaulted to monolithic (single PUT) blob uploads for ACR destinations.

  2. Copy flow ignores adapter capability — The copy flow (flow/copy.go) only checked the policy-level CopyByChunk flag and ignored the destination adapter's SupportedCopyByChunk declaration. Even if an adapter declared chunk support, users had to manually enable it per replication policy — a setting that is not exposed in the UI for non-Harbor registries.

  3. req.ContentLength not set in PushBlobChunk() — The registry client set Content-Length via req.Header.Set(), but Go's HTTP client ignores manually set Content-Length headers. It uses the req.ContentLength struct field to decide between sending a Content-Length header or using Transfer-Encoding: chunked. Without setting req.ContentLength, Go sent blob chunks with HTTP chunked transfer encoding, which Azure ACR rejects with CONTENT_LENGTH_INVALID (400).

Changes

File Change
src/pkg/reg/adapter/azurecr/adapter.go Set SupportedCopyByChunk: true in Info()
src/controller/replication/flow/copy.go Auto-enable chunked upload when destination adapter declares SupportedCopyByChunk support
src/pkg/registry/client.go Set req.ContentLength in PushBlobChunk() to ensure proper Content-Length header

Testing

  • Unit tests — Updated existing tests for the adapter and copy flow, including a new test case (TestCopyByChunkAutoEnabled) that validates auto-enabling chunked uploads based on adapter capability.
  • Integration test — Added integration_test.go that performs a real 25MB chunked blob upload (5 × 5MB chunks) against a live Azure Container Registry instance. The test is skipped unless ACR_URL, ACR_USER, and ACR_PASS environment variables are set.
=== RUN   TestChunkedUploadToACR/chunked_upload_succeeds
    Uploading chunk 0-5242879 (5242880 bytes, last=false)
    OK - new location length: 328
    Uploading chunk 5242880-10485759 (5242880 bytes, last=false)
    OK - new location length: 336
    Uploading chunk 10485760-15728639 (5242880 bytes, last=false)
    OK - new location length: 336
    Uploading chunk 15728640-20971519 (5242880 bytes, last=false)
    OK - new location length: 336
    Uploading chunk 20971520-26214399 (5242880 bytes, last=true)
    OK - new location length: 106
    === SUCCESS: Chunked upload to Azure ACR works! ===
--- PASS: TestChunkedUploadToACR (6.09s)

How to run integration test

ACR_URL=https://yourregistry.azurecr.io ACR_USER=admin ACR_PASS=secret \
  go test -v -run TestChunkedUploadToACR ./src/pkg/reg/adapter/azurecr/...

When replicating images from Harbor to Azure Container Registry, blob
uploads larger than ~20MB fail with HTTP 413 (Request Entity Too Large).

Root cause analysis revealed three issues:

1. The Azure ACR adapter did not declare SupportedCopyByChunk in its
   Info() method, so the replication flow always used monolithic
   (single PUT) uploads for ACR destinations.

2. The copy flow only checked the policy-level CopyByChunk flag and
   ignored the destination adapter's declared capability. This meant
   even if an adapter declared chunk support, users had to manually
   enable it per replication policy.

3. The registry client's PushBlobChunk() set Content-Length via
   req.Header.Set() but did not set req.ContentLength on the
   http.Request struct. Go's HTTP client ignores manually set
   Content-Length headers and relies on req.ContentLength to decide
   between Content-Length and Transfer-Encoding: chunked. As a result,
   Go sent the request with Transfer-Encoding: chunked (HTTP-level),
   which Azure ACR rejects with CONTENT_LENGTH_INVALID.

Changes:
- Set SupportedCopyByChunk: true in the Azure ACR adapter's Info()
- Auto-enable chunked upload in the copy flow when the destination
  adapter declares support via SupportedCopyByChunk
- Set req.ContentLength in PushBlobChunk() to ensure Go sends a
  proper Content-Length header instead of Transfer-Encoding: chunked

Validated with an integration test that performs a real 25MB chunked
blob upload (5 x 5MB chunks) against a live Azure ACR instance.

Fixes goharbor#23015
Related: goharbor#21640

Signed-off-by: Chrysostomos Kolovos <xrkolovos@egritosgroup.gr>
@xrkolovos xrkolovos requested a review from a team as a code owner March 19, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure ACR replication adapter does not support chunked/resumable blob uploads (413 error)

4 participants