Infinite Pagination in cloudwatchlogs.NewGetLogEventsPaginator()

[osmontero]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

When paginating through CloudWatch Logs events via cloudwatchlogs.NewGetLogEventsPaginator(), the end of the log stream is not properly detected. Instead, the final page of the result set is continually requested in an infinite loop.

The problem is that the GetLogEvents API signals the end of the stream by setting nextForwardToken or nextBackwardToken to the same as the requested value (and not null, like most other paginated APIs). GetLogEventsPaginator evaluates if the nextToken is null instead of evaluating if it is equal, resulting in an infinity loop.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

HasMorePages() must return false if there are no more pages

Current Behavior

HasMorePages() always returns true

Reproduction Steps

paginator := cloudwatchlogs.NewGetLogEventsPaginator(cwl, &cloudwatchlogs.GetLogEventsInput{
  LogGroupName:  aws.String(logGroup),
  LogStreamName: aws.String(stream),
  StartTime:     aws.Int64(startTime.Unix() * 1000),
  EndTime:       aws.Int64(endTime.Unix() * 1000),
  StartFromHead: aws.Bool(true),
})

for paginator.HasMorePages() {
  page, err := paginator.NextPage(ctx)
  if err != nil {
    return nil, catcher.Error("cannot get logs", err, nil)
  }
  
  for _, event := range page.Events {
    transformedLogs = append(transformedLogs, *event.Message)
  }
}

More details can be found here: main.go

Possible Solution

evaluate if the nextForwardToken is equal to the requested params.NextToken

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.32.8
github.com/aws/aws-sdk-go-v2/config v1.28.11
github.com/aws/aws-sdk-go-v2/credentials v1.17.52

Compiler and Version used

go1.23.4 linux/amd64

Operating System and version

Debian Bookworm

[osmontero]

As a workaround I have created a variable to store the last nextForwardToken and evaluate if it is equal to the NextPage nextForwardToken

var lastToken string

for paginator.HasMorePages() {
	page, err := paginator.NextPage(ctx)
	if err != nil {
		return nil, catcher.Error("cannot get logs", err, nil)
	}
				
	if lastToken == *page.NextForwardToken {
		break
	}
				
	lastToken = *page.NextForwardToken
				
	for _, event := range page.Events {
		transformedLogs = append(transformedLogs, *event.Message)
	}
}

[lucix-aws]

You checked "potential regression" - did this not happen in a previous release?

[osmontero]

You checked "potential regression" - did this not happen in a previous release?

I had it working with the SDK v1, started happening when I switched to use v2

[osmontero]

It looks like that setting StopOnDuplicateToken to true resolves the problem, but if it is the common behavior of the logs API, it is not better to set this flag as true by default? Actually, I think this flag is not necessary at all

paginator := cloudwatchlogs.NewGetLogEventsPaginator(cwl, &cloudwatchlogs.GetLogEventsInput{
	LogGroupName:  aws.String(logGroup),
	LogStreamName: aws.String(stream),
	StartTime:     aws.Int64(startTime.Unix() * 1000),
	EndTime:       aws.Int64(endTime.Unix() * 1000),
	StartFromHead: aws.Bool(true),
}, func(options *cloudwatchlogs.GetLogEventsPaginatorOptions) {
	options.StopOnDuplicateToken = true
	options.Limit = 10000
})

[lucix-aws]

The service client SDKs here are generated from API models (note that the file you modified in your PR has a "generated by x" comment at the top).

So while this may be the case for cloudwatchlogs APIs across the board, that's not information we have in the modeled paginator (here, or anywhere else, to my knowledge). So, from an SDK perspective, there's not any action we can take here. Explicitly setting StopOnDuplicateToken is the way forward here for the time being.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.