S3 list_objects_v2 paginator MaxItems only counts keys (Contents) not prefixes (CommonPrefixes)

[bsmedberg-xometry]

Describe the bug

When using boto3 to iterate an S3 bucket with a Delimiter, MaxItems only counts the keys, not the prefixes. So if you have a bucket with only prefixes, MaxItems will never stop searching and may take unbounded time.

Steps to reproduce

Set up a bucket with 20000 keys of the form result1/results.txt ... result20000/results.txt

Run this code:

import boto3
s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')
for result in paginator.paginate(Bucket='mybucket', Delimiter='/', PaginationConfig={'MaxItems': 2000}):
    for prefix in result.get('CommonPrefixes', []):
        print("prefix {}".format(prefix['Prefix']))
    for key in result.get('Contents', []):
        print("key {}".format(key['Key'])

Expected behavior
The above program should return a maximum of 2000 keys. It actually returns all 20,000 keys, because MaxItems doesn't count prefixes.

[swetashre]

@bsmedberg-xometry - I am able to reproduce the issue. Thank you for pointing it out. Marking this as bug.

[swetashre]

@bsmedberg-xometry - After some digging into the code base i found that this is the expected behavior. When it is not a first request(means starting token is not included) then we are only considering the first result key response to truncate. So in this case we are only considering to truncate the response of Contents not CommonPrefixes.

Can you please confirm whether its the number of keys you are getting , more than MaxItems or the number of commonprefixes ?

[bsmedberg-xometry]

In this case, there will be nothing in Contents ever, there will only be CommonPrefixes.

And the problem is not the truncation: the problem is that even after getting 2000 CommonPrefixes, it keeps making calls forever.

I have worked around this locally by doing something like this:

import boto3
s3 = boto3.client('s3')
paginator = s3.get_paginator('list_objects_v2')
found_keys = 0
found_prefixes = 0
for result in paginator.paginate(Bucket='mybucket', Delimiter='/', PaginationConfig={'MaxItems': 2000}):
    found_prefixes += len(result.get('CommonPrefixes', []))
    found_keys += len(result.get('Contents', []))
    if found_prefixes + found_keys > 2000:
        break # stop iterating here to prevent eternal iteration

However I don't believe that this is or should be the expected behavior of the boto3 paginator. If this is is the expected behavior of the paginator, then the paginator docs need to be updated to warn of this behavior.

[shepazon]

While I agree with @bsmedberg-xometry (hey man, what's up? 😁) that this behavior seems really wrong (what good is the MaxItems option if the thing will just ignore it?), I am looking into what I can do to the documentation to explain this if in fact it's truly intended to work this way.

That said, I think it would be very helpful to be able to explain why it works this way. Can someone explain that to me, so I can write this up properly?

[bsmedberg-xometry]

Or even better: add a behavior to the S3 listobjects API call that actually does the right thing. I'm happy for this to turn into an S3 feature request instead of a boto problem if the problem is actually that S3 doesn't provide a logical pagination API.