Skip to content

Feat(Storage): Enable full object checksum validation on JSON path #8825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
thiyaguk09 wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Feat(Storage): Enable full object checksum validation on JSON path #8825

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
053370b
Feat(Storage): Enable full object checksum validation on JSON path
thiyaguk09 Dec 19, 2025
69be82e
bug fix
thiyaguk09 Dec 19, 2025
db306f4
bug fix
thiyaguk09 Dec 19, 2025
6513fc4
Feat(core): ensure X-Goog-Hash is only sent on final upload segments
thiyaguk09 Mar 20, 2026
2ac9aad
lint fix
thiyaguk09 Mar 20, 2026
1e627f4
chore: code refactor
thiyaguk09 Mar 23, 2026
34f2f0b
Merge branch 'main' into feat/enable-full-checksum-validation
thiyaguk09 Apr 1, 2026
4895c98
feat(core): support all custom headers and fix final chunk hash logic
thiyaguk09 Apr 1, 2026
ddfc6e7
refactor(storage): clean up hash logic and leverage autovivification
thiyaguk09 Apr 1, 2026
e24a1e3
Lint fix
thiyaguk09 Apr 1, 2026
d4dd640
Merge branch 'main' into feat/enable-full-checksum-validation
thiyaguk09 Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Core/src/Upload/MultipartUploader.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ private function prepareRequest()
$headers['Content-Length'] = $size;
}

$customHeaders = $this->requestOptions['restOptions']['headers'] ?? [];
$headers = array_merge($headers, $customHeaders);

return new Request(
'POST',
$this->uri,
Expand Down
7 changes: 7 additions & 0 deletions Core/src/Upload/ResumableUploader.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,13 @@ public function upload()
'Content-Range' => "bytes $rangeStart-$rangeEnd/$size",
];

if ($size !== '*' && ($rangeEnd + 1) == (int) $size) {
$customHeaders = $this->requestOptions['restOptions']['headers'] ?? [];
if (isset($customHeaders['X-Goog-Hash'])) {
$headers['X-Goog-Hash'] = $customHeaders['X-Goog-Hash'];
Copy link
Copy Markdown
Contributor

@bshaffer bshaffer Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we are only setting this one header instead of supporting all custom headers?

Copy link
Copy Markdown
Contributor Author

@thiyaguk09 thiyaguk09 Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. The intent was to isolate the X-Goog-Hash to the final chunk to avoid intermediate validation errors, but this implementation accidentally drops other custom headers. I will refactor this to merge all $customHeaders while specifically unsetting X-Goog-Hash only on non-final chunks.

}
}

$request = new Request(
'PUT',
$resumeUri,
Expand Down
11 changes: 10 additions & 1 deletion Core/src/Upload/StreamableUploader.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,12 @@ public function upload($writeSize = null)
return [];
}

$isFinalRequest = ($writeSize === null);

// find or create the resumeUri
$resumeUri = $this->getResumeUri();

if ($writeSize) {
if ($writeSize !== null) {
$rangeEnd = $this->rangeStart + $writeSize - 1;
$data = $this->data->read($writeSize);
} else {
Expand All @@ -62,6 +64,13 @@ public function upload($writeSize = null)
'Content-Range' => "bytes {$this->rangeStart}-$rangeEnd/*"
];

if ($isFinalRequest) {
$customHeaders = $this->requestOptions['restOptions']['headers'] ?? [];
if (isset($customHeaders['X-Goog-Hash'])) {
$headers['X-Goog-Hash'] = $customHeaders['X-Goog-Hash'];
Copy link
Copy Markdown
Contributor

@bshaffer bshaffer Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question here - is there a reason we're only supporting the one custom header instead of supporting all custom headers?

Copy link
Copy Markdown
Contributor Author

@thiyaguk09 thiyaguk09 Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent was to isolate the X-Goog-Hash to the final chunk to avoid intermediate validation errors, but this implementation accidentally drops other custom headers. I will refactor this to merge all $customHeaders while specifically unsetting X-Goog-Hash only on non-final chunks.

}
}

$request = new Request(
'PUT',
$resumeUri,
Expand Down
42 changes: 42 additions & 0 deletions Core/tests/Unit/Upload/MultipartUploaderTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,48 @@ public function testUploadsAsyncData()
$actualPromise->wait()
);
}

public function testUploadsWithCustomHeaders()
{
$customHeaders = [
'X-Goog-Custom-Header' => 'custom-value',
'User-Agent' => 'custom-ua'
];

$requestWrapper = $this->prophesize(RequestWrapper::class);
$stream = Utils::streamFor('abcd');
$successBody = '{"canI":"kickIt"}';
$response = new Response(200, [], $successBody);

$requestWrapper->send(
Argument::that(function (RequestInterface $request) use ($customHeaders) {
foreach ($customHeaders as $key => $value) {
if ($request->getHeaderLine($key) !== $value) {
return false;
}
}

$contentType = $request->getHeaderLine('Content-Type');
return str_contains($contentType, 'multipart/related')
&& str_contains($contentType, 'boundary=boundary');
}),
Argument::type('array')
)->willReturn($response);

$uploader = new MultipartUploader(
$requestWrapper->reveal(),
$stream,
'http://www.example.com',
[
'restOptions' => [
'headers' => $customHeaders
]
]
);

$this->assertEquals(json_decode($successBody, true), $uploader->upload());
}

/**
* @dataProvider streamSizes
*/
Expand Down
113 changes: 113 additions & 0 deletions Core/tests/Unit/Upload/ResumableUploaderTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,119 @@ public function testRetryOptionsPassing()
$this->assertTrue($retryListenerCalled);
}

public function testUploadSendsGoogHashOnFinalChunk()
{
$hashValue = 'crc32c=abc123';
$resumeUri = 'http://some-resume-uri.example.com';

$this->requestWrapper->send(
Argument::which('getMethod', 'POST'),
Argument::type('array')
)->willReturn(new Response(200, ['Location' => $resumeUri]));

$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) use ($hashValue) {
return $request->getHeaderLine('X-Goog-Hash') === $hashValue;
}),
Argument::type('array')
)->willReturn(new Response(200, [], $this->successBody));

$uploader = new ResumableUploader(
$this->requestWrapper->reveal(),
$this->stream,
'http://www.example.com',
[
'restOptions' => [
'headers' => ['X-Goog-Hash' => $hashValue]
]
]
);

$this->assertEquals(json_decode($this->successBody, true), $uploader->upload());
}

public function testUploadDoesNotSendGoogHashOnIntermediateChunk()
{
$hashValue = 'crc32c=abc123';
$resumeUri = 'http://some-resume-uri.example.com';

$this->requestWrapper->send(
Argument::which('getMethod', 'POST'),
Argument::type('array')
)->willReturn(new Response(200, ['Location' => $resumeUri]));

$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) {
return $request->getHeaderLine('Content-Range') === 'bytes 0-1/4'
&& !$request->hasHeader('X-Goog-Hash');
}),
Argument::type('array')
)->willReturn(new Response(308, ['Range' => 'bytes 0-1']));

$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) {
return $request->getHeaderLine('Content-Range') === 'bytes 2-3/4';
}),
Argument::type('array')
)->willReturn(new Response(200, [], $this->successBody));

$uploader = new ResumableUploader(
$this->requestWrapper->reveal(),
$this->stream, // size 4
'http://www.example.com',
[
'chunkSize' => 2, // Force multi-chunk upload
'restOptions' => [
'headers' => ['X-Goog-Hash' => $hashValue]
]
]
);

$this->assertEquals(json_decode($this->successBody, true), $uploader->upload());
}

public function testGoogHashOnlyOnFinalChunkOfMultiChunkUpload()
{
$hashValue = 'crc32c=abc123';
$resumeUri = 'http://some-resume-uri.example.com';
$this->stream = Utils::streamFor('01234567'); // 8 bytes total

$this->requestWrapper->send(
Argument::which('getMethod', 'POST'),
Argument::type('array')
)->willReturn(new Response(200, ['Location' => $resumeUri]));

$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) {
return $request->getHeaderLine('Content-Range') === 'bytes 0-3/8'
&& !$request->hasHeader('X-Goog-Hash');
}),
Argument::type('array')
)->willReturn(new Response(308, ['Range' => 'bytes 0-3']));

$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) use ($hashValue) {
return $request->getHeaderLine('Content-Range') === 'bytes 4-7/8'
&& $request->getHeaderLine('X-Goog-Hash') === $hashValue;
}),
Argument::type('array')
)->willReturn(new Response(200, [], $this->successBody));

$uploader = new ResumableUploader(
$this->requestWrapper->reveal(),
$this->stream,
'http://www.example.com',
[
'chunkSize' => 4,
'restOptions' => [
'headers' => ['X-Goog-Hash' => $hashValue]
]
]
);

$this->assertEquals(json_decode($this->successBody, true), $uploader->upload());
}

public function testThrowsExceptionWhenAttemptsAsyncUpload()
{
$this->expectException(GoogleException::class);
Expand Down
109 changes: 109 additions & 0 deletions Core/tests/Unit/Upload/StreamableUploaderTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,115 @@ public function testLastChunkSendsCorrectHeaders()
$uploader->upload();
}

public function testUploadWithCustomGoogHashHeader()
{
$hashValue = 'md5=abc123';
$resumeUri = 'http://some-resume-uri.example.com';

$resumeResponse = new Response(200, ['Location' => $resumeUri]);
$this->requestWrapper->send(
Argument::type(RequestInterface::class),
Argument::any()
)->willReturn($resumeResponse)->shouldBeCalled();

$uploadResponse = new Response(200, ['Location' => $resumeUri], $this->successBody);
$this->requestWrapper->send(
Argument::that(function ($request) use ($resumeUri, $hashValue) {
return (string) $request->getUri() === $resumeUri
&& $request->getHeaderLine('X-Goog-Hash') === $hashValue;
}),
Argument::any()
)->willReturn($uploadResponse)->shouldBeCalled();

$uploader = new StreamableUploader(
$this->requestWrapper->reveal(),
$this->stream,
'http://www.example.com',
[
'restOptions' => [
'headers' => [
'X-Goog-Hash' => $hashValue,
'Other-Header' => 'should-be-ignored'
]
]
]
);

$this->stream->write("some data");

$this->assertEquals(json_decode($this->successBody, true), $uploader->upload());
}

public function testUploadDoesNotSendGoogHashWhenConditionNotMet()
{
$hashValue = 'md5=abc123';
$resumeUri = 'http://some-resume-uri.example.com';

$resumeResponse = new Response(200, ['Location' => $resumeUri]);
$this->requestWrapper->send(
Argument::which('getMethod', 'POST'),
Argument::type('array')
)->willReturn($resumeResponse);

$uploadResponse = new Response(200, [], $this->successBody);
$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) {
return !$request->hasHeader('X-Goog-Hash');
}),
Argument::type('array')
)->willReturn($uploadResponse);

$uploader = new StreamableUploader(
$this->requestWrapper->reveal(),
$this->stream,
'http://www.example.com',
[
'restOptions' => [
'headers' => ['X-Goog-Hash' => $hashValue]
]
]
);

$this->stream->write("0123456789ABCDEF");

$this->assertEquals(json_decode($this->successBody, true), $uploader->upload(16));
}

public function testUploadSendsGoogHashOnFinalStep()
{
$hashValue = 'md5=finalHash';
$resumeUri = 'http://some-resume-uri.example.com';

$resumeResponse = new Response(200, ['Location' => $resumeUri]);
$this->requestWrapper->send(
Argument::which('getMethod', 'POST'),
Argument::type('array')
)->willReturn($resumeResponse)->shouldBeCalled();

$uploadResponse = new Response(200, [], $this->successBody);
$this->requestWrapper->send(
Argument::that(function (RequestInterface $request) use ($hashValue) {
return $request->getHeaderLine('X-Goog-Hash') === $hashValue;
}),
Argument::type('array')
)->willReturn($uploadResponse)->shouldBeCalled();

$uploader = new StreamableUploader(
$this->requestWrapper->reveal(),
$this->stream,
'http://www.example.com',
[
'restOptions' => [
'headers' => ['X-Goog-Hash' => $hashValue]
]
]
);

$this->stream->write("final data");

$this->assertEquals(json_decode($this->successBody, true), $uploader->upload());
}

public function testThrowsExceptionWhenAttemptsAsyncUpload()
{
$this->expectException(GoogleException::class);
Expand Down
Loading
Loading