Describe the feature
Enabling ChecksumMode when calling getObject increases the response time.
This happens as the response stream is consumed for validating the checksum during the API call.
This was done in #5043 to ensure that checksum validation error is thrown during the API call and an error is thrown. However, since we return a stream, the checksum validation should be delayed till stream is consumed by the callee.
Use Case
Test case
import { S3 } from "@aws-sdk/client-s3"; // v3.654.0
import { equal } from "assert";
const client = new S3();
const Bucket = "test-checksum-mode"; // Replace with your test bucket name.
const Key = "hello-world.txt";
const Body = "Hello World\n".repeat(100_000); // File of size ~1 MB.
const ChecksumAlgorithm = "CRC32";
// The putObject call be commented out for subsequent calls for benchmarking getObject.
await client.putObject({ Bucket, Key, Body, ChecksumAlgorithm });
console.time("getObject");
const response = await client.getObject({
Bucket,
Key,
// ChecksumMode: "ENABLED",
});
console.timeEnd("getObject");
equal(Body, await response.Body.transformToString());API call times compared with file size/repeatations
| File size |
Repeat # for putObject |
getObject without ChecksumMode |
getObject with ChecksumMode |
| ~1 MB |
100_000 |
27ms |
85ms |
| ~10 MB |
1_000_000 |
28ms |
450ms |
| ~100 MB |
10_000_000 |
39ms |
3676ms |
The numbers will differ depending on your network speed, but the difference will remain
Proposed Solution
Write a checksum stream wrapper class which consumes the stream for validation. only when end user consumes it.
Other Information
No response
Acknowledgements
SDK version used
3.654.0
Environment details (OS name and version, etc.)
v20.10.0
Describe the feature
Enabling ChecksumMode when calling getObject increases the response time.
This happens as the response stream is consumed for validating the checksum during the API call.
This was done in #5043 to ensure that checksum validation error is thrown during the API call and an error is thrown. However, since we return a stream, the checksum validation should be delayed till stream is consumed by the callee.
Use Case
Test case
API call times compared with file size/repeatations
The numbers will differ depending on your network speed, but the difference will remain
Proposed Solution
Write a checksum stream wrapper class which consumes the stream for validation. only when end user consumes it.
Other Information
No response
Acknowledgements
SDK version used
3.654.0
Environment details (OS name and version, etc.)
v20.10.0