fix(paginate): detect pagination tokens that cycle, not just repeat

[Adityaj0]

Issue # (if applicable)

fixes #3778

Reason for this change

PageIterator.__iter__ guards against non-terminating pagination by comparing each new next_token against previous_next_token, a single-slot memory of the immediately preceding token:

previous_next_token = None
...
if (
    previous_next_token is not None
    and previous_next_token == next_token
):
    raise PaginationError(...)
self._inject_token_into_kwargs(current_kwargs, next_token)
previous_next_token = next_token

This catches a token repeating on two consecutive pages, but misses a longer cycle (A, B, A, B, ...): at every step next_token differs from the token immediately before it, so the check never fires and pagination runs forever, re-fetching the same pages. This is exactly the failure mode the check exists to prevent — a service handing back a token that makes no forward progress — it just doesn't generalize past a one-token lookback.

Description of changes

Replaced the single previous_next_token slot with a list of every next_token seen so far (seen_next_tokens), and check membership in that list instead of equality with only the last one:

seen_next_tokens = []
...
if next_token in seen_next_tokens:
    raise PaginationError(...)
self._inject_token_into_kwargs(current_kwargs, next_token)
seen_next_tokens.append(next_token)

next_token is a dict, so a set isn't usable directly (unhashable); a list with in keeps the same semantics as the original equality check while covering cycles of any length. Pagination depth is bounded in practice, so the linear scan is not a concern.

Describe any new or updated permissions being added

None.

Description of how you validated changes

Added test_exception_raised_if_next_token_cycles to tests/unit/test_paginate.py, using the same self.paginator/self.method fixture as the existing test_exception_raised_if_same_next_token, but with a token sequence token1, token2, token1, token2 that never repeats consecutively. It fails (hangs/never raises before this fix would need a manual timeout) on main and passes with this change.

$ python -m pytest tests/unit/test_paginate.py -v -k "same_next_token or cycles"
test_exception_raised_if_same_next_token PASSED
test_exception_raised_if_next_token_cycles PASSED

$ python -m pytest tests/unit/test_paginate.py -q
75 passed

$ python -m pytest tests/functional/test_paginate.py -q
9 passed

Also manually verified with a standalone repro driving PageIterator directly against a fake method cycling NextToken between two values — hangs indefinitely on main, raises PaginationError with this change. And confirmed the existing consecutive-duplicate case still raises correctly (no regression to the original behavior).

Backwards compatibility

The only externally-visible change is that pagination which previously ran forever (a hang, effectively unusable) now raises PaginationError instead. No currently-terminating pagination changes behavior — a cycling token is never legitimate forward progress, so this only converts an infinite hang into a clear error.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[Adityaj0]

Updated this to fix a regression I found in my own original approach: seen_next_tokens was a list (since next_token is a dict and unhashable), and next_token in seen_next_tokens is an O(k) linear scan run once per page — O(n²) total across a full pagination.

Measured with a synthetic page loop:

list-based check:  16,000 pages -> 1.83s   |  200,000 pages -> didn't finish in 120s
set-based check:   16,000 pages -> 0.004s  |  200,000 pages -> 0.16s

That's a real problem for the workloads this touches — S3 listings, DynamoDB scans, CloudWatch Logs, etc. can legitimately run into tens/hundreds of thousands of pages, so the original fix would have traded an unobserved hang for a guaranteed slowdown on every large, perfectly healthy pagination.

Fixed by hashing next_token into a frozenset(next_token.items()) and storing those in a set instead, restoring O(1) membership checks (with a repr-based fallback if a token value is ever itself unhashable). All existing and new unit/functional tests still pass; re-verified the original cycling repro still raises PaginationError correctly with this change.