It should be possible to download a blob with content-encoding while keeping the content-encoding

[kosta]

Is your feature request related to a problem? Please describe.

I put a blob onto an Azure Storage Container with content-encoding gzip. In python, I call something like

# plain text is 'hello from gzip'
gzipped_data = b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xcaH\xcd\xc9\xc9WH+\xca\xcfUH\xaf\xca,\x00\x00\x00\x00\xff\xff\x03\x00d\xaa\x8e\xb5\x0f\x00\x00\x00'
azure_blob = BlobClient(...)
content_settings = ContentSettings(content_encoding='gzip', ...)
azure_blob.upload_blob(data=gzipped_data, length=len(gzipped_data), content_settings=content_settings)

Now I have a Blob with the given Content-encoding. Downloading it via e.g. curl will yield the raw, compressed data. (Your browser will most likely decompress it for you)

When I download it again:

azure_blob = BlobClient(...)
blob_data = azure_blob.download_blob()
downloaded = blob_data.read()

Now, downloaded is b'hello from gzip', but I expected gzipped_data from the first code block.

Describe the solution you'd like

BlobClient.download_blob() should have the option to keep the content-encoding, e.g. by passing decompress=False. This seems to be in line with existing code in the Azure SDK.

Describe alternatives you've considered

For my use case, e.g. 1:1 copying from A to B, I do not see an alternative.

Additional context

The code is almost there. The StreamDownloadGenerator, which is being used under the hood, has code for it already. I just needs to be passed decompress=False.

However, in BlobOperations.download() the decompress kwarg is not passed. I tested locally to add _decompress = kwargs.pop("decompress", True) to the top of that function, and then pass decompress=_decompress to both instances of response.stream_download(). It seems to work.

BlobOperations is generated code. Please advise perform the modification above through code generation. I tried looking at https://github.com/Azure/azure-rest-api-specs/ but did not know where to start.

[pvaneck]

Hey, thanks for the feedback. I'll tag some folks, and we'll investigate as soon as possible.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[kosta]

I'm currently using this code as a workaround, but it does not work for all auth methods and also it's very fragile:

        url = f"{self.account_url}/{self.bucket_name}/{quote(full_path)}"
        azure_http_request = HttpRequest(
            method="GET",
            url=url,
            headers={
                "x-ms-version": "2023-11-03",
                "x-ms-date": format_date_time(time()),
                "x-ms-client-request-id": str(uuid.uuid1()),
            },
        )
        if offset != 0:
            azure_http_request.headers["Range"] = f"bytes={offset}-"
            azure_http_request.headers["byte_range"] = f"bytes={offset}-"  # dont ask 🙄
        azure_pipeline_request = PipelineRequest(
            http_request=azure_http_request, context=PipelineContext(transport=azure_blob._pipeline._transport)
        )
        assert azure_blob._credential_policy
        azure_blob._credential_policy.on_request(azure_pipeline_request)
        assert "Authorization" in azure_http_request.headers

        response = self.http.request(  # type: ignore
            method="GET",
            url=url,
            headers=azure_http_request.headers,
            decode_content=False,  # keep compression by default
            preload_content=False,  # 🙈
        )

[jalauzon-msft]

Hi @kosta, thanks for the report and detailed investigation. This is something that has come up before and something we would like to add. We just have not had time to figure out how to get things to play nice in the generated code. We will try and look at this again soon and update back on this post.