Skip to content

RcloneListDirIter: unify list chunk size for agent and backend #4134

Open
@Michal-Leszczynski

Description

@Michal-Leszczynski

Basically all backends chunk dir entries when listing. E.g. s3 has the MaxKeys param which is set to 1000 by default.
We also have our own chunking mechanism implemented:

type listJSONEncoder struct {
enc *jsonEncoder
buf []*rcops.ListJSONItem
maxItems int
started bool
}

In the past we used the same chunk size in the backend queries and in our own chunking mechanism, which made sense.
Starting from commit 43230ec, we don't set the backend chunk size anymore. This means that we use the default chunk size for the backend (e.g. 1000 for s3), and a different value for our own chunking mechanism:

const defaultListEncoderMaxItems = 200

I believe that it was mistakenly forgotten during refactor and that we should align those chunk sizes.
Let's go with the intentional 200.

Metadata

Metadata

Labels

backupbugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions