Datasets stored in S3 may be compressed as .tar.gz archives. Add automatic decompression during loader pull.
- Detect
.tar.gz / .tgz files and decompress + extract in-stream during download
- Avoid writing the compressed archive to disk
- Update progress output to show compressed vs decompressed sizes
Datasets stored in S3 may be compressed as
.tar.gzarchives. Add automatic decompression duringloader pull..tar.gz/.tgzfiles and decompress + extract in-stream during download