gRPC: Input/Output error while writing repeatedly over the existing file

[raj-prince]

Describe the issue
Got Input/output error while performing write benchmark via fio over a mount with --client-protocol=grpc. I am trying to re-write over 10MiB file again and again.

Here is the GCSFuse error logs,

time="08/12/2024 12:37:17.753653" severity=INFO message="Retrying for the error: rpc error: code = ResourceExhausted desc = The object gcs-grpc-team-liutimothy-bucket-fuse-002/10M/10M_write.0.0 exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429."
time="08/12/2024 12:37:18.554019" severity=INFO message="Retrying for the error: rpc error: code = ResourceExhausted desc = The object gcs-grpc-team-liutimothy-bucket-fuse-002/10M/10M_write.0.0 exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429."
time="08/12/2024 12:37:19.998119" severity=INFO message="Retrying for the error: rpc error: code = ResourceExhausted desc = The object gcs-grpc-team-liutimothy-bucket-fuse-002/10M/10M_write.0.0 exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429."
time="08/12/2024 12:37:23.496142" severity=ERROR message="SyncFile: input/output error, FileInode.Sync: SyncObject: create: CreateObject: error in closing writer : EOF"
time="08/12/2024 12:37:23.496188" severity=ERROR message="fuse: *fuseops.SyncFileOp error: input/output error"

System & Version (please complete the following information):

• OS: All
• Platform All
• Version - v2.0.1 onwards

SLO:
We strive to respond to all bug reports within 24 business hours provided the information mentioned above is included.