`modal volume get` can silently create sparse or truncated files after short block downloads

[jasondavies]

Summary

modal volume get can exit successfully while producing corrupted local files. In a real bulk download, some files contained multi-megabyte sparse/zero-filled regions and another file was silently truncated.

The corruption is consistent with an HTTP response for an 8 MiB Volume block ending early with a successful status. The Python client accepts however many bytes arrive without validating them against VolumeGetFile2Response.len. Later blocks are written at fixed 8 MiB offsets, so a short intermediate block leaves an unwritten sparse region. A short final block produces a truncated file.

Environment

• Modal client: 1.5.3
• Platform: Linux
• Command: modal volume get
• Volume path: rectangle-free-data-v2/rect8x8-production/solve
• Dataset: 1,024 files, approximately 104 GiB total
• The local destination directory existed before the command, so this was not caused by treating a nonexistent directory as a filename.

Reproduction

I ran:

mkdir -p /tmp/modal-repro/existing
modal volume get \
  rectangle-free-data-v2 \
  rect8x8-production/solve \
  /tmp/modal-repro/existing

The download ran for approximately 17 minutes and exited with status 0:

✓ Finished downloading files to local!

All 1,024 filenames existed afterward, but exact-size and sparse-file validation found:

• 14 same-size files containing sparse, zero-filled regions
• 1 silently truncated file
• 15 corrupted files total (approximately 1.46% of the download)

The sparse files were:

s0111.orbits
s0243.orbits
s0289.orbits
s0301.orbits
s0421.orbits
s0488.orbits
s0504.orbits
s0521.orbits
s0548.orbits
s0608.orbits
s0637.orbits
s0902.orbits
s0968.orbits
s0975.orbits

Some representative zero runs were:

File	Zero-run start	Zero-run length	Zero-run end
s0111.orbits	53,477,376	5,242,884	58,720,260 (56 MiB + 4)
s0421.orbits	3,080,192	5,308,420	8,388,612 (8 MiB + 4)
s0488.orbits	78,118,912	5,767,172	83,886,084 (80 MiB + 4)
s0975.orbits	11,206,656	5,570,564	16,777,220 (16 MiB + 4)

The extra four bytes are consistent with this file format's 20-byte header followed by 16-byte records. At the filesystem level, the holes themselves ended exactly at the 8 MiB boundaries.

The truncated file was:

s0792.orbits
remote size: 112,680,404 bytes
bulk-download size: 111,869,952 bytes
shortfall: 810,452 bytes

The local truncated size is 13 full 8 MiB blocks plus 2,818,048 bytes, which is also one of the partial-block cutoff offsets observed in the sparse files.

Verification that the Volume data is intact

I downloaded each of the 15 affected files again, individually, into a separate directory:

modal volume get \
  rectangle-free-data-v2 \
  rect8x8-production/solve/s0111.orbits \
  /tmp/modal-repro/verification

The same procedure was repeated for all affected paths. Every individual download:

• completed successfully;
• had the expected remote size;
• was fully allocated rather than sparse; and
• had a different SHA-256 hash from its corrupted bulk-download copy.

This rules out corrupted stored Volume files. Since the bulk command itself exited successfully, it also rules out an interrupted download as the cause of this reproduction.

Relevant client behavior

VolumeGetFile2Response provides the number of bytes expected:

message VolumeGetFile2Response {
  repeated string get_urls = 1;
  uint64 size = 2;
  uint64 start = 3;
  uint64 len = 4; // number of bytes returned
}

However, _read_file_into_fileobj() does not use response.len for integrity validation. Each URL is downloaded independently, and a block is considered successful after any successful-status response reaches EOF:

block_start_pos = start_pos + idx * BLOCK_SIZE

async with session.get(url) as get_response:
    await _raise_on_block_response_error(get_response)
    async for chunk in get_response.content.iter_any():
        # write bytes at block_start_pos + ...

Afterward, the client only sums the bytes it actually received:

total_size = sum(await asyncio.gather(*coros))
fileobj.seek(start_pos + total_size)
return total_size

There is no comparison between total_size and response.len. Because blocks are placed at fixed 8 MiB offsets, receiving only a prefix of an intermediate block followed by a complete later block creates exactly the sparse pattern observed here.

The streaming Volume.read_file() path likewise yields all returned block bodies without a final comparison to response.len.

Relevant source locations:

• py/modal/volume.py: Volume.read_file() and _read_file_into_fileobj()
• py/modal/cli/_download.py: destination files are opened directly with "wb"
• modal_proto/api.proto: VolumeGetFile2Response.len

Expected behavior

The client should never report a successful download unless the exact expected number of bytes was received. A short block response should be retried or reported as an integrity error.

The CLI should also avoid publishing a partially written file under its final name. On failure, either the existing destination should remain intact or no final destination file should exist.

Suggested fixes

1. Validate downloaded bytes against VolumeGetFile2Response.len in both download paths.
2. Where possible, derive and validate the expected length of every block so a short response can be retried by the existing per-block retry mechanism.
3. Treat an unexpected URL count or aggregate byte count as an integrity failure.
4. Cancel and await sibling block tasks when any block fails.
5. Download CLI files to temporary files in the destination directory, then atomically rename them only after validation succeeds.
6. Preserve an existing file until its replacement has downloaded and validated successfully.

Suggested regression tests

• An intermediate block URL returns a short 200 body and a later block succeeds.
• The final block URL returns a short 200 body.
• A chunked 200 response ends early without an HTTP-status error.
• The response contains too few block URLs for response.len.
• Volume.read_file() rejects an aggregate length smaller than response.len.
• read_file_into_fileobj() rejects an aggregate length smaller than response.len.
• The CLI does not leave a corrupt file under the requested final name after failure.
• --force preserves the previous valid file until the replacement succeeds.

Additional note

This demonstrates the client-side integrity failure but does not identify why the block service, proxy, or CDN returned cleanly terminated short bodies. Capturing that initiating fault would require HTTP-level telemetry such as the requested range, response headers, expected block length, received length, and request ID. Regardless of the upstream cause, the client has enough protocol information to detect and reject the corrupt result.

Version

1.5.3

App ID

No response

[jasondavies]

One additional detail: this affects v2 Volumes. My reproductions were using modal volume get against v2 Volume data.