write_documents failing for bigger messages

[akshaykarle]

When running write_documents I get the following error:

grpc RpcError: [query], <_MultiThreadedRendezvous: StatusCode.RESOURCE_EXHAUSTED, grpc: trying to send message larger than max (69004574 vs. 67108864)>, <Time:{'RPC start': '2023-10-16 11:46:26.850932', 'gRPC error': '2023-10-16 11:46:38.551411'}>

I noticed in _create_collection we set the max_length. is there a reason we need to set that? Is that possibly causing this to fail? If so, can we remove and/or allow overriding it?

[xiaofan-luan]

the batch size is too large.

try to reduce the batch to less than 4M, currently it paased the default 64M setting

[akshaykarle]

Okay, that's what I thought. Will try reducing the batch_size and see if it works. Thanks for your quick reply. Will get back to you once I've tested it and we can possibly close this of

[apohllo]

It's a pitty this is not a parametr I can change in the call. Changing the library's code is the only way to achieve that?

[apohllo]

Ok. So it is a parameter, my bad. But... get_all_documents does not use a batch size, so if the size of docs is larger than the message, there's the same error.