Open
Description
Basically all backends chunk dir entries when listing. E.g. s3 has the MaxKeys
param which is set to 1000 by default.
We also have our own chunking mechanism implemented:
scylla-manager/pkg/rclone/rcserver/list.go
Lines 12 to 17 in b97d1cf
In the past we used the same chunk size in the backend queries and in our own chunking mechanism, which made sense.
Starting from commit 43230ec, we don't set the backend chunk size anymore. This means that we use the default chunk size for the backend (e.g. 1000 for s3), and a different value for our own chunking mechanism:
I believe that it was mistakenly forgotten during refactor and that we should align those chunk sizes.
Let's go with the intentional 200.