feat(storage): add support for streaming uploads by brianquinlan · Pull Request #219 · googleapis/google-cloud-dart

brianquinlan · 2026-03-26T22:58:50Z

Fixes #180

Supports uploading GCS objects using a StreamSink<List<int>>.

This PR trades performance and correctness for simplicity. Future PRs (see the TODO in the code) will fix the correctness problems.

Calls to add accumulate data in memory until close is called, which causes the data to be sent to GCS.
Calls to addStream accumulate data into 256KiB chunks (the minimum GCS chunk size) and then sends that data to GCS.

It is possible for GCS to partially reject a write. For example, you could send 100MiB to GCS in a single HTTP request and GCS could indicate that 50MiB was successfully stored. That scenario is not currently supported and the upload will fail in that case.

gemini-code-assist

Code Review

This pull request introduces the ResumableUploadSink class and a corresponding uploadObjectFromSink method in the Storage client to support resumable uploads to Google Cloud Storage using a StreamSink. The implementation includes internal buffering to meet GCS chunk size requirements and is accompanied by a comprehensive test suite. Review feedback highlights a significant performance improvement opportunity in the _flush method to reduce memory allocations and copies. Other suggestions focus on improving the clarity of error messages when concurrent operations are attempted, ensuring consistent use of StateError, and correcting a misleading test name.

pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

pkgs/google_cloud_storage/test/upload_object_from_sink_test.dart

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

brianquinlan · 2026-03-27T13:51:15Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces the ResumableUploadSink class and the uploadObjectFromSink method to support resumable uploads to Google Cloud Storage using a StreamSink. While the implementation provides a solid foundation, several improvements are needed for production readiness. Key feedback includes addressing a critical bug where the upload offset is updated optimistically without verifying the server's received byte range, optimizing buffer management to reduce memory allocations and copies, and updating method signatures to return Future for better interface compliance. Additionally, the documentation should be updated to warn users about potential memory issues when using add() for large uploads.

gemini-code-assist · 2026-03-27T13:53:09Z

pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

+    // TODO(https://github.com/googleapis/google-cloud-dart/issues/218):
+    // Check the "range" headers to determine if any data must be resent.
+    _nextExpectedByte = newEnd;


The current implementation optimistically updates _nextExpectedByte without verifying the actual number of bytes received by the server. In a resumable upload, it's possible for the server to receive only a partial chunk.

The 308 Resume Incomplete response from Google Cloud Storage includes a Range header (e.g., Range: bytes=0-262143) that specifies which bytes have been successfully stored. You must parse this header to determine the correct next byte offset.

Failing to do so can lead to data corruption if a chunk is only partially uploaded. While there's a TODO for this, I'm flagging it as critical because it affects the core correctness of the upload functionality.

gemini-code-assist · 2026-03-27T13:53:09Z

pkgs/google_cloud_storage/lib/src/client.dart

+  /// Creates or updates the content of a [Google Cloud Storage object][] using
+  /// a [StreamSink].
+  ///
+  /// If [metadata] is non-null, it will be used as the object's metadata. If
+  /// [metadata] is `null` or `metadata.contentType` is `null`, the content type
+  /// will be `'application/octet-stream'`. If `metadata.name` does not
+  /// match [name], a [BadRequestException] is thrown.
+  ///


The documentation should clarify the behavior of the returned StreamSink. Specifically, calls to sink.add() will buffer all data in memory until sink.close() is invoked. This can lead to excessive memory usage and potentially an OutOfMemoryError when uploading large files.

Please add a note advising users to prefer sink.addStream() for large uploads to take advantage of chunked streaming and avoid high memory consumption.

For example, you could add:

/// **Important**: When using the returned [ResumableUploadSink], be aware that /// calls to `sink.add()` will buffer all data in memory until `sink.close()` is /// called. For large uploads, this can lead to high memory usage. It is /// recommended to use `sink.addStream()` for large files to stream data in /// chunks and avoid buffering the entire content in memory.

gemini-code-assist · 2026-03-27T13:53:09Z

pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

+    if (requiredCapacity > _writeBuffer.length) {
+      var newSize = _writeBuffer.length * 2;
+      if (newSize < requiredCapacity) {
+        newSize = requiredCapacity;
+      }
+      newSize =
+          ((newSize + _minWriteSize - 1) ~/ _minWriteSize) * _minWriteSize;
+
+      final newBuffer = Uint8List(newSize)
+        ..setRange(0, _writeBufferSize, _writeBuffer);
+      _writeBuffer = newBuffer;


The current buffer resizing logic in _addToBuffer creates a new Uint8List and copies all existing data whenever the buffer capacity is exceeded. This can be inefficient for performance, especially when dealing with many small add() calls.

For a more performant implementation, consider using BytesBuilder. It is designed for this use case and can often avoid reallocating and copying the entire buffer on each growth by managing an internal list of chunks.

gemini-code-assist · 2026-03-27T13:53:09Z

pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

+    final flushPoint = _largestWriteSize(_writeBufferSize);
+    if (flushPoint == 0) return;
+    final flushBuffer = _writeBuffer;
+    _writeBuffer = _writeBuffer.sublist(flushPoint);


Using _writeBuffer.sublist(flushPoint) creates a new Uint8List and copies the remaining data after a flush. This copy operation can be a performance bottleneck, especially if the remaining part of the buffer is large.

A more efficient approach would be to avoid creating a new list. You could, for example, move the remaining data to the beginning of the same buffer using _writeBuffer.setRange(0, _writeBufferSize - flushPoint, _writeBuffer, flushPoint) and then update the size. This would avoid an allocation and a copy.

gemini-code-assist · 2026-03-27T13:53:09Z

pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

+  }
+
+  @override
+  Future<dynamic> addStream(Stream<List<int>> stream) async {


The addStream method is defined to return Future<dynamic>. According to the StreamSink interface, this method should return Future<void>.

Using Future<void> improves type safety and adheres to the interface contract. Please update the return type.

Suggested change

Future<dynamic> addStream(Stream<List<int>> stream) async {

Future<void> addStream(Stream<List<int>> stream) async {

gemini-code-assist · 2026-03-27T13:53:09Z

pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

+  }
+
+  @override
+  Future<dynamic> close() async {


The close method is defined to return Future<dynamic>. According to the StreamSink interface, this method should return Future<void>.

Using Future<void> improves type safety and adheres to the interface contract. Please update the return type.

Suggested change

Future<dynamic> close() async {

Future<void> close() async {

natebosch · 2026-03-27T23:59:44Z

pkgs/google_cloud_storage/lib/src/client.dart

+  /// See [API reference docs](https://docs.cloud.google.com/storage/docs/performing-resumable-uploads#chunked-upload).
+  ///
+  /// [Google Cloud Storage object]: https://docs.cloud.google.com/storage/docs/json_api/v1/objects
+  ResumableUploadSink uploadObjectFromSink(


[nit] I don't think we need to expose the ResumableUploadSink type, I don't see any API surface it adds.

Suggested change

ResumableUploadSink uploadObjectFromSink(

StreamSink<List<int>> uploadObjectFromSink(

I'm modeling ResumableUploadSink after the C++ ObjectWriteStream class.

Eventually, it will support the properties required to resume an existing upload e.g. https://github.com/googleapis/google-cloud-cpp/blob/fdbec33ff40b1d6ae9b740cbacf969be4141255a/google/cloud/storage/object_write_stream.h#L273

natebosch · 2026-03-28T00:00:25Z

pkgs/google_cloud_storage/lib/google_cloud_storage.dart

 export 'src/object_metadata.dart';
 export 'src/object_metadata_patch_builder.dart' show ObjectMetadataPatchBuilder;
 export 'src/project_team.dart';
+export 'src/resumeable_upload.dart' show ResumableUploadSink;


Do we need to export this?

Suggested change

export 'src/resumeable_upload.dart' show ResumableUploadSink;

natebosch · 2026-03-28T00:02:22Z

pkgs/google_cloud_storage/test/upload_object_from_sink_test.dart

+  final small = randomUint8List(100);
+  final large = randomUint8List(5_000_000);


Should we use the seed argument for consistent test data?

Do you think I should? I hope that it doesn't matter.

If there does end up being a failure for some sets of inputs having an unknown seed can make reproductions harder.

On the other hand, using a fixed seed loses some of the fuzzing we get otherwise and we don't currently have a nice API for using changing but reproducible random seeds from the test runner. If we had dart-lang/test#1572 I'd argue for using it. I just checked and printOnFailure doesn't work in setUpAll which is the other approach I wanted to suggest dart-lang/test#2620

I'm happy to stick with a random seed for now and we can revisit if it ever becomes a problem or we have a nice universally recommended pattern.

Yeah, I was doing this for fuzzing purposes. dart-lang/test#1572 is a good idea!

natebosch · 2026-03-28T00:04:41Z

pkgs/google_cloud_storage/lib/src/client.dart

+  /// See [API reference docs](https://docs.cloud.google.com/storage/docs/performing-resumable-uploads#chunked-upload).
+  ///
+  /// [Google Cloud Storage object]: https://docs.cloud.google.com/storage/docs/json_api/v1/objects
+  ResumableUploadSink uploadObjectFromSink(


Should we take a Stream argument instead of returning a StreamSink? It would let us return a Future<ObjectMetadata> like the other method.

That's a good idea. I want to leave the existing interface to support resumable uploads. But adding uploadObjectFromStream seems like a good addition (and trivial to implement).

I think we could still support resumable uploads. I might be overlooking some detail, but I think returning a StreamSink and taking a Stream argument are interdefinable APIs.

If we implement as uploadObjectFromSink as you have here, then we can define

Future<void> uploadObjectFromStream( String bucket, String name, Stream<List<int>> stream, { ObjectMetadata? metadata, }) async { final sink = uploadObjectFromSink(bucket, name, metadata: metadata); try { await sink.addStream(stream); } finally { await sink.close(); } }

Alternatively if we implement as uploadObjectFromStream then the current API can wrap it.

StreamSink<List<int>> uploadObjectFromSink( String bucket, String name, { ObjectMetadata? metadata, }) { final controller = StreamController<List<int>>(); uploadObjectFromStream( bucket, name, controller.stream, metadata: metadata ).ignore(); return controller.sink; }

In either case the underlying sink can use the resumable implementation so I think we only need to pick which signature we like the best. If the resumable sink has extra members of StreamSink they wouldn't be equivalent, but I don't recall it defining anything public that wasn't an override.

I missed that the resumable sink will get new public members.

Having both APIs SGTM

brianquinlan added 9 commits March 25, 2026 18:10

feat(storage): add support for streaming uploads

f66fac9

add works

98dcbf5

More stuff working

d5b4a6c

Fix

905abf2

More work

2df941a

Fox

ac1c3a2

fix

12e146f

Fix

5ba8eed

fixes

3749958

gemini-code-assist bot reviewed Mar 26, 2026

View reviewed changes

brianquinlan and others added 6 commits March 26, 2026 16:03

Update pkgs/google_cloud_storage/test/upload_object_from_sink_test.dart

f988885

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update pkgs/google_cloud_storage/lib/src/resumeable_upload.dart

95c2af5

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fixes

126d38a

Merge branch 'main' into sink_upload

58e45ba

fix

8feddd4

Comments.

e3bcb99

brianquinlan requested review from natebosch and sigurdm March 27, 2026 00:33

gemini-code-assist bot reviewed Mar 27, 2026

View reviewed changes

Merge branch 'main' into sink_upload

7feedfd

natebosch reviewed Mar 28, 2026

View reviewed changes

Merge branch 'main' into sink_upload

739d272

	Future<dynamic> addStream(Stream<List<int>> stream) async {
	Future<void> addStream(Stream<List<int>> stream) async {

	ResumableUploadSink uploadObjectFromSink(
	StreamSink<List<int>> uploadObjectFromSink(

		final small = randomUint8List(100);
		final large = randomUint8List(5_000_000);

Conversation

brianquinlan commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brianquinlan commented Mar 27, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

brianquinlan commented Mar 26, 2026 •

edited

Loading