Skip to content

fix: resolve sources of CI flakiness and IAM propagation issues - #740

Open
apeabody wants to merge 1 commit into
mainfrom
fix/ci
Open

fix: resolve sources of CI flakiness and IAM propagation issues#740
apeabody wants to merge 1 commit into
mainfrom
fix/ci

Conversation

@apeabody

Copy link
Copy Markdown
Collaborator
  • Add explicit time_sleep resources and DAG dependencies for IAM role bindings across root test setup and harnesses
  • Eliminate duplicate IAM role assignment causing 409 concurrent policy modification conflicts
  • Wire time_sleep.wait_propagation into deployment pipeline modules and examples
  • Fix build directory path and add IAM propagation delay for binary authorization image creation
  • Fix service network peering sleep dependency in private workerpool harness
  • Expand retryable transient error patterns in testutils for IAM, rate limits, and network errors
  • Harden integration test polling loops to tolerate transient container startup and kubectl connection errors

@apeabody
apeabody requested review from a team, sleighton2022 and yliaog as code owners August 25, 2026 18:36

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves deployment and test stability by introducing propagation delays (using time_sleep resources) for IAM permissions and network peerings, updating resource dependencies, and expanding retryable error patterns in integration tests. The review feedback suggests avoiding printing full pod logs during polling in hello_world_e2e_test.go to prevent excessively verbose CI logs, proposing a simpler progress message instead.

} else {
return false, fmt.Errorf("Unable to get hello world container running.")
}
t.Logf("Application starting up, current logs: %s", logs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Printing the entire pod log output on every polling iteration can lead to extremely verbose test logs and potential log truncation in CI environments. Consider logging a simple progress message instead of the full log content during each retry.

Suggested change
t.Logf("Application starting up, current logs: %s", logs)
t.Log("Application starting up, waiting for 'Hello world!' in logs...")

- Add explicit time_sleep resources and DAG dependencies for IAM role bindings across root test setup and harnesses
- Eliminate duplicate IAM role assignment causing 409 concurrent policy modification conflicts
- Wire time_sleep.wait_propagation into deployment pipeline modules and examples
- Fix build directory path and add IAM propagation delay for binary authorization image creation
- Fix service network peering sleep dependency in private workerpool harness
- Expand retryable transient error patterns in testutils for IAM, rate limits, and network errors
- Harden integration test polling loops to tolerate transient container startup and kubectl connection errors
- Fix config sync policy path defaulting in multicluster discovery integration test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant