Skip to content

🌱 e2e: fix panic when dumping CloudTrail logs #5747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
k8s-ci-robot merged 1 commit into from
Nov 17, 2025
Merged

🌱 e2e: fix panic when dumping CloudTrail logs #5747

k8s-ci-robot merged 1 commit into from
Nov 17, 2025

Conversation

@chrischdi
Copy link
Member

@chrischdi chrischdi commented Nov 17, 2025

What type of PR is this?

/kind flake

What this PR does / why we need it:

Periodic e2e main shows this panic:

https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-provider-aws#periodic-e2e-main

Example: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-provider-aws-e2e/1990331610909642752

------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/suites/unmanaged/unmanaged_suite_test.go:57
------------------------------
[SynchronizedAfterSuite] [PANICKED] [1751.284 seconds]
[SynchronizedAfterSuite] 
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/suites/unmanaged/unmanaged_suite_test.go:57
  Timeline >>
  INFO: Creating log watcher for controller capa-system/capa-controller-manager, pod capa-controller-manager-6dbb98f7b5-mplkd, container manager
  Folder created for eks clusters: "/logs/artifacts/clusters/bootstrap/aws-resources"
  Couldn't get AWS CloudTrail events: err=operation error CloudTrail: LookupEvents, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: e57af8dc-1865-447d-8509-c13cc407926f, api error ThrottlingException: Rate exceeded
  [PANICKED] in [SynchronizedAfterSuite] - /usr/local/go/src/runtime/panic.go:262 @ 11/17/25 09:17:28.535
  << Timeline
  [PANICKED] Test Panicked
  In [SynchronizedAfterSuite] at: /usr/local/go/src/runtime/panic.go:262 @ 11/17/25 09:17:28.535
attachment
  runtime error: invalid memory address or nil pointer dereference
  Full Stack Trace
    sigs.k8s.io/cluster-api-provider-aws/v2/test/e2e/shared.DumpCloudTrailEvents(0xc001089400)
    	/home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/shared/aws.go:923 +0x345
    sigs.k8s.io/cluster-api-provider-aws/v2/test/e2e/shared.Node1AfterSuite(0xc001089400)
    	/home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/shared/suite.go:287 +0x68
    sigs.k8s.io/cluster-api-provider-aws/v2/test/e2e/suites/unmanaged.init.func12()
    	/home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/suites/unmanaged/unmanaged_suite_test.go:62 +0x1a
------------------------------
[ReportAfterSuite] PASSED [0.028 seconds]
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
------------------------------
Summarizing 1 Failure:
  [PANICKED!] [SynchronizedAfterSuite] 
  /usr/local/go/src/runtime/panic.go:262

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

  • squashed commits
  • includes documentation
  • includes emoji in title
  • adds unit tests
  • adds or updates e2e tests

Release note:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/flake Categorizes issue or PR as related to a flaky test. labels Nov 17, 2025
@chrischdi
Copy link
Member Author

chrischdi commented Nov 17, 2025

/test help

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 17, 2025
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 17, 2025

@chrischdi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build

/test pull-cluster-api-provider-aws-build-docker

/test pull-cluster-api-provider-aws-e2e-blocking

/test pull-cluster-api-provider-aws-test

/test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-apidiff-main

/test pull-cluster-api-provider-aws-e2e

/test pull-cluster-api-provider-aws-e2e-clusterclass

/test pull-cluster-api-provider-aws-e2e-conformance

/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts

/test pull-cluster-api-provider-aws-e2e-eks

/test pull-cluster-api-provider-aws-e2e-eks-gc

/test pull-cluster-api-provider-aws-e2e-eks-testing

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-apidiff-main

pull-cluster-api-provider-aws-build

pull-cluster-api-provider-aws-build-docker

pull-cluster-api-provider-aws-e2e-blocking

pull-cluster-api-provider-aws-test

pull-cluster-api-provider-aws-verify
Details

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Nov 17, 2025
@chrischdi
Copy link
Member Author

chrischdi commented Nov 17, 2025

Wanna test if there are other e2e's also perma-failing:

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-clusterclass
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e-eks-gc

@chrischdi
Copy link
Member Author

chrischdi commented Nov 17, 2025

Regarding: pull-cluster-api-provider-aws-e2e-eks-gc

This test was removed in #5348 .

We should remove the presubmit for this :-)

@chrischdi chrischdi mentioned this pull request Nov 17, 2025
Merged
damdo
damdo reviewed Nov 17, 2025
View reviewed changes
Copy link
Member

@damdo damdo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

/assign @nrb @richardcase @AndiDog @dlipovetsky @Ankitasw

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 17, 2025
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 17, 2025

LGTM label has been added.

DetailsGit tree hash: 79fdcb019915e917adcff79d1c2e6aa882186ef0

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e-conformance

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 17, 2025 •
edited
Loading

@chrischdi: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cluster-api-provider-aws-e2e-eks-gc 694e191 link false /test pull-cluster-api-provider-aws-e2e-eks-gc
pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts 694e191 link false /test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/cherry-pick release-2.9

@k8s-infra-cherrypick-robot
Copy link

k8s-infra-cherrypick-robot commented Nov 17, 2025

@damdo: once the present PR merges, I will cherry-pick it on top of release-2.9 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-2.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e

@nrb
Copy link
Contributor

nrb commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts

@damdo
Copy link
Member

damdo commented Nov 17, 2025

/test pull-cluster-api-provider-aws-e2e

@nrb
Copy link
Contributor

nrb commented Nov 17, 2025

/approve

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 17, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nrb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 17, 2025
@k8s-ci-robot k8s-ci-robot merged commit c0ef9f4 into kubernetes-sigs:main Nov 17, 2025
26 of 28 checks passed
@k8s-infra-cherrypick-robot
Copy link

k8s-infra-cherrypick-robot commented Nov 17, 2025

@damdo: new pull request created: #5749

Details

In response to this:

/cherry-pick release-2.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot k8s-infra-cherrypick-robot mentioned this pull request Nov 17, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@damdo damdo damdo left review comments

@Ankitasw Ankitasw Awaiting requested review from Ankitasw

@cnmcavoy cnmcavoy Awaiting requested review from cnmcavoy

Assignees

@richardcase richardcase

@nrb nrb

@AndiDog AndiDog

@dlipovetsky dlipovetsky

@damdo damdo

@Ankitasw Ankitasw

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@chrischdi @k8s-ci-robot @damdo @k8s-infra-cherrypick-robot @nrb @richardcase @AndiDog @dlipovetsky @Ankitasw