remote cache ResourceExhausted error

[kevin-oliveira-zocdoc]

Describe the bug
When trying to use pants on a nodejs project with a large package.json/node_modules folder, I'm seeing the following warning close to when the yarn install finishes, which looks related to when node_modules tries to get remote cached:

Failed to write to remote cache (1 occurrences so far):
ResourceExhausted: "grpc: received message larger than max (7819978 vs. 4194304)"

This happened on pants 2.24.

I noticed a similar issue/fix (#20674 and #20708) was pushed to 2.26, so I did try 2.26.0.dev7 but I still got the error, however with a slight change on the message size:

Failed to write to remote cache (1 occurrences so far):
ResourceExhausted: "grpc: received message larger than max (4251997 vs. 4194304)"

The message size looks closer to the limit accepted by the grpc server, but still higher.
When I re-ran the pants command I see different message sizes, e.g. a second ran returned:

Failed to write to remote cache (1 occurrences so far):
ResourceExhausted: "grpc: received message larger than max (4252192 vs. 4194304)"

So I'm assuming the batched requests previous implemented to fix this issue is occurring but the message size sent is somehow still higher than 4194304.

Pants version
2.26.0.dev7

OS
Linux

[tgolsson]

Thanks for the report! The message size heuristic is probably too conservative. I will have a look again if we can make it more accurate. Or just fudge-factor it another 5-10%.