Skip to content

Fix flaky tests for azsdk-cli by removing non-deterministic behavior#47

Closed
Copilot wants to merge 2 commits intomainfrom
copilot/fix-46
Closed

Fix flaky tests for azsdk-cli by removing non-deterministic behavior#47
Copilot wants to merge 2 commits intomainfrom
copilot/fix-46

Conversation

Copy link
Copy Markdown

Copilot AI commented Jul 11, 2025

This PR fixes two flaky tests in the azsdk-cli test suite that were causing inconsistent failures in the CI pipeline:

Issues Fixed

1. TestSimpleFailCase - Random Exit Code Generation

The test was using new Random().Next(2) to generate a random expected exit code (0 or 1), but the actual implementation always returns exit code 1. This created a race condition where the test would:

  • Pass when random generated 1 (matching actual behavior)
  • Fail when random generated 0 (not matching actual behavior)

Before:

var expectedExitCode = new Random().Next(2);
var expected = $"[ERROR] RESPONDING TO 'HI. MY NAME IS' with FAIL: {expectedExitCode}";

After:

var expected = "[ERROR] RESPONDING TO 'HI. MY NAME IS' with FAIL: 1";

2. TestWindows - Incorrect Expected Output

The test expected "Result: nul" but the actual output was "Result: null", causing consistent failures on Windows platforms.

Before:

var expected = @"
Message: RESPONDING TO 'HI. MY NAME IS' with SUCCESS: 0
Result: nul
Duration: 1ms".TrimStart();

After:

var expected = @"
Message: RESPONDING TO 'HI. MY NAME IS' with SUCCESS: 0
Result: null
Duration: 1ms".TrimStart();

Testing

  • Verified tests now pass consistently across multiple runs
  • Confirmed all existing tests in the suite continue to pass
  • Changes are minimal and surgical, maintaining the original test intent

The fixes eliminate non-deterministic behavior while preserving the core functionality being tested.

Fixes #46.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

…xpected output

Co-authored-by: benbp <1020379+benbp@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix flaky tests for azsdk cli Fix flaky tests for azsdk-cli by removing non-deterministic behavior Jul 11, 2025
Copilot finished work on behalf of benbp July 11, 2025 21:58
Copilot AI requested a review from benbp July 11, 2025 21:58
@benbp benbp closed this Oct 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix flaky tests for azsdk cli

2 participants