Module google-cloud-storage: retrieving an object's size at rest via the data plane

[mmukada]

Hi, I am integrating google-cloud-storage and struggling to design testable logic for retrieving an object's size at rest. The SDK exposes this via the control plane, which works but isn't supported by standard test tools like fake-gcs-server, as they focus on the data plane.

I tried using the data plane instead - hoping for a mechanism to request with alt=json or similar, but the SDK always sends media requests (alt=media), so that path doesn't work. My fallback was to trigger a download without consuming the stream and read the size from the response headers. However, this relies on the x-goog-stored-content-length header, which, according to Gemini and ChatGPT, may be absent in some cases.

Do you have any suggestions for a reliable and testable approach?

[coryan]

Hi, I am integrating google-cloud-storage and struggling to design testable logic for retrieving an object's size at rest. The SDK exposes this via the control plane, which works but isn't supported by standard test tools like fake-gcs-server, as they focus on the data plane.

Does fake-gcs-server implement any metadata operations? Can you list or delete objects? Can you get the object metadata? Calling StorageControl::get_object should just use a http request without the alt=media query parameter. If fake-gcs-server supports that operation your program should work.

My fallback was to trigger a download without consuming the stream and read the size from the response headers. However, this relies on the x-goog-stored-content-length header, which, according to Gemini and ChatGPT, may be absent in some cases.

Hmm... not in my experience, but the public documentation is ambiguous (let me ask internally). If that is the case, then you have a different problem beyond testing: the application won't be able to get the object size in production either, no?

Do you have any suggestions for a reliable and testable approach?

For testable: can you use mocks? You can mock both the control and data plane operations in the Rust client library.

For reliable: if you must have the object size, query the object metadata. That may require fixing fake-gcs-server if it does not support that operation.

[mmukada]

Thanks for getting back @coryan

I used AI to analyse the codebase, which returned with the following findings:

storage_control.get_object() uses gRPC (over HTTP/2) via Tonic.

Evidence in the codebase

• File: src/storage/src/generated/gapic/transport.rs. The get_object transport implementation inserts a Tonic gRPC method annotation and targets the gRPC service/method path:

e.insert(tonic::GrpcMethod::new(
    "google.storage.v2.Storage",
    "GetObject",
));
let path = http::uri::PathAndQuery::from_static("/google.storage.v2.Storage/GetObject");

It then executes and decodes via gRPC helpers:

self.inner
    .execute( /* ... */ )
    .await
    .and_then(gaxi::grpc::to_gax_response::<TR, crate::model::Object>)

These show the call is dispatched as a Tonic gRPC unary RPC to the google.storage.v2.Storage/GetObject method, not via REST/JSON.

I have checked and can confirm that fake-gcs-server correctly handles metadata requests via REST/JSON, by interpreting the alt request parameter (or the absense thereof) appropriately.

With regards to the x-goog-stored-content-length header, GCP provided this in the test I conducted, which aligns with the SDK. However, I am not sure whether it is guaranteed in all media downloads as expected by the SDK. fake-gcs-server communicates the size via the Content-Length header - which appears to be ommitted in the SDK. I am a little unsure what the GCP standard is, and am happy patching fake-gcs-server, if need be, once I have confirmation.

Mocks work fine, but emulators are more reliable and better suited for local end-to-end testing.

[coryan]

storage_control.get_object() uses gRPC (over HTTP/2) via Tonic.

You are right. I completely forgot about that.

With regards to the x-goog-stored-content-length header, GCP provided this in the test I conducted, which aligns with the SDK. However, I am not sure whether it is guaranteed in all media downloads as expected by the SDK. fake-gcs-server communicates the size via the Content-Length header

That is the size of the download. Which can be smaller than the object (say if you are downloading a range), or larger (if the object is being decompressed on the fly). It may also be omitted.

happy patching fake-gcs-server, if need be, once I have confirmation.

I checked internally and looked at the code. The x-goog-stored-content-length header is always present.

[mmukada]

I checked internally and looked at the code. The x-goog-stored-content-length header is always present.

Thanks, @coryan. This is helpful. I'll work on integrating x-goog-stored-content-length into fake-gcs-server, which, as you noted, better aligns with metadata.size. This will enable me to take the alternative approach of inspecting the header without consuming the response.