Backport of paginator: fix tokenizer comparison of composite index and ID into release/1.10.x

[hc-github-team-nomad-core]

Backport

This PR is auto-generated from #25792 to be assessed for backporting due to the inclusion of the label backport/1.10.x.

The below text is copied from the body of the original PR.

---

Description

The CreateIndexAndIDTokenizer creates a composite token by
combining the create index value and ID from the object with
a .. Tokens are then compared lexicographically. The comparison
is appropriate for the ID segment of the token, but it is not for
the create index segement. Since the create index values are stored
with numeric ordering, using a lexicographical comparison can cause
unexpected results.

For example, when comparing the token 12.object-id to 102.object-id
the result will show 12.object-id being greater. This is the
correct comparison but it is incorrect for the intention of the token.
With the knowledge of the composition of the token, the response
should be that 12.object-id is less.

The unexpected behavior can be seen when performing lists (like listing
allocations). The behavior is encountered inconsistently due to
two requirements which must be met:

1. Create index values with a large enough span (ex: 12 and 102)
2. Correct per page value to get a "bad" next token (ex: prefix with 102)

To prevent the unexpected behavior, the target token is split
and the components are used individually to compare against the
object.

Testing & Reproduction steps

1. Create a pageable number of allocations and space creation over a long enough timespan for the CreateIndex to reach three digits
2. List allocations and paginate the response such that the NextToken value is at the start of the three digit CreateIndex values
3. Request next page of results and view entries from start of list

Example

A simple script was used to cycle through page sizes, listing allocations and checking for duplicate objects. It displays where the comparison results in unexpected behavior:

Completed loop 138 (size: 3 next_token: "88.aef58062-8cd8-e5ba-33e2-f3ebe0054b0b")
  -> Item ID: 88.aef58062-8cd8-e5ba-33e2-f3ebe0054b0b
  -> Item ID: 101.96f23439-8921-572e-70fd-71a724a56b85
  -> Item ID: 117.5eb20d56-59eb-91e4-c92e-fc17bd714f3b
Completed loop 139 (size: 3 next_token: "117.d0ae8856-7863-80f1-abaa-52c05172449f")
  -> Item ID: 12.00438448-6b52-024d-1efd-ed1ace4147b4
ERROR - Item ID (12.00438448-6b52-024d-1efd-ed1ace4147b4) already seen on loop number 1, current index: 0

Links

#25435

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

---

Overview of commits

• a69baee

[hashicorp-cla-app]

All committers have signed the CLA.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}