Skip to content

S3 Backend - if file is gzip'd, file is always opened in binary mode #1526

@john-parton

Description

@john-parton

For the S3 backend, if a file is gzipped/compressed, open will always open the file in binary mode.

Here's the rough code path:

self._decompress_file is called here:

if self._storage.gzip and self.obj.content_encoding == "gzip":
self._file = self._decompress_file(mode=self._mode, file=self._file)

Defined on this mixin:

def _decompress_file(self, mode, file, mtime=0.0):
return GzipFile(mode=mode, fileobj=file, mtime=mtime)

Consulting the python docs for GzipFile: https://docs.python.org/3/library/gzip.html#gzip.GzipFile

"Note that the file is always opened in binary mode. To open a compressed file in text mode, use open() (or wrap your GzipFile with an io.TextIOWrapper)."

Solution is to either use gzip.open or explicitly wrap the GzipFile in a TextIOWrapper

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions