fix: enable chunked blob upload for Azure ACR replication#23016
Open
xrkolovos wants to merge 1 commit intogoharbor:mainfrom
Open
fix: enable chunked blob upload for Azure ACR replication#23016xrkolovos wants to merge 1 commit intogoharbor:mainfrom
xrkolovos wants to merge 1 commit intogoharbor:mainfrom
Conversation
When replicating images from Harbor to Azure Container Registry, blob uploads larger than ~20MB fail with HTTP 413 (Request Entity Too Large). Root cause analysis revealed three issues: 1. The Azure ACR adapter did not declare SupportedCopyByChunk in its Info() method, so the replication flow always used monolithic (single PUT) uploads for ACR destinations. 2. The copy flow only checked the policy-level CopyByChunk flag and ignored the destination adapter's declared capability. This meant even if an adapter declared chunk support, users had to manually enable it per replication policy. 3. The registry client's PushBlobChunk() set Content-Length via req.Header.Set() but did not set req.ContentLength on the http.Request struct. Go's HTTP client ignores manually set Content-Length headers and relies on req.ContentLength to decide between Content-Length and Transfer-Encoding: chunked. As a result, Go sent the request with Transfer-Encoding: chunked (HTTP-level), which Azure ACR rejects with CONTENT_LENGTH_INVALID. Changes: - Set SupportedCopyByChunk: true in the Azure ACR adapter's Info() - Auto-enable chunked upload in the copy flow when the destination adapter declares support via SupportedCopyByChunk - Set req.ContentLength in PushBlobChunk() to ensure Go sends a proper Content-Length header instead of Transfer-Encoding: chunked Validated with an integration test that performs a real 25MB chunked blob upload (5 x 5MB chunks) against a live Azure ACR instance. Fixes goharbor#23015 Related: goharbor#21640 Signed-off-by: Chrysostomos Kolovos <xrkolovos@egritosgroup.gr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #23015 — Azure ACR replication fails with HTTP 413 for blob layers larger than ~20MB.
Related: #21640
Root cause
Three issues were identified:
Missing adapter capability declaration — The Azure ACR adapter's
Info()method did not setSupportedCopyByChunk: true, so the replication flow always defaulted to monolithic (single PUT) blob uploads for ACR destinations.Copy flow ignores adapter capability — The copy flow (
flow/copy.go) only checked the policy-levelCopyByChunkflag and ignored the destination adapter'sSupportedCopyByChunkdeclaration. Even if an adapter declared chunk support, users had to manually enable it per replication policy — a setting that is not exposed in the UI for non-Harbor registries.req.ContentLengthnot set inPushBlobChunk()— The registry client setContent-Lengthviareq.Header.Set(), but Go's HTTP client ignores manually setContent-Lengthheaders. It uses thereq.ContentLengthstruct field to decide between sending aContent-Lengthheader or usingTransfer-Encoding: chunked. Without settingreq.ContentLength, Go sent blob chunks with HTTP chunked transfer encoding, which Azure ACR rejects withCONTENT_LENGTH_INVALID(400).Changes
src/pkg/reg/adapter/azurecr/adapter.goSupportedCopyByChunk: trueinInfo()src/controller/replication/flow/copy.goSupportedCopyByChunksupportsrc/pkg/registry/client.goreq.ContentLengthinPushBlobChunk()to ensure proper Content-Length headerTesting
TestCopyByChunkAutoEnabled) that validates auto-enabling chunked uploads based on adapter capability.integration_test.gothat performs a real 25MB chunked blob upload (5 × 5MB chunks) against a live Azure Container Registry instance. The test is skipped unlessACR_URL,ACR_USER, andACR_PASSenvironment variables are set.How to run integration test
ACR_URL=https://yourregistry.azurecr.io ACR_USER=admin ACR_PASS=secret \ go test -v -run TestChunkedUploadToACR ./src/pkg/reg/adapter/azurecr/...