Skip to content

[flaky] Fix flaky tests caused by port binding#149

Merged
cendhu merged 2 commits into
hyperledger:mainfrom
liran-funaro:fix-flaky-test-ports
Oct 30, 2025
Merged

[flaky] Fix flaky tests caused by port binding#149
cendhu merged 2 commits into
hyperledger:mainfrom
liran-funaro:fix-flaky-test-ports

Conversation

@liran-funaro

Copy link
Copy Markdown
Contributor

Type of change

  • Test update

Description

  • [ServerConfig] Add retry mechanism when binding to a predefined port (c.Listener()).
  • [TestBroadcastDeliver] Use the above retry mechanism to ensure the port is not used by other tests.

Related issues

Comment thread utils/connection/server_util.go Outdated

var err error
var listener net.Listener
if ctx == nil || c.Endpoint.Port == 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not intuitive that when the ctx is nil, retry till successful binding is not executed. I notice all callers to pass the ctx in the current code. Do we need this check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover code. Good catch!

// It stores the listener object internally to be reused on subsequent calls to Listener().
func (c *ServerConfig) PreAllocateListener() (net.Listener, error) {
listener, err := c.Listener()
listener, err := c.Listener(context.Background())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we plan to pass the ctx as an argument to this PreallocateListener() in the future?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. PreallocateListener should only be used with port=0. If such attempt fails, there is no reason to retry.

t.Log("One server down")
servers[2].Servers[0].Stop()
waitUntilGrpcServerIsDown(ctx, t, &servers[2].Configs[0].Endpoint)
listener1 := holdPort(ctx, t, servers[2].Configs[0])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Earlier, we only checked whether the server was down or not. Now, we create another fake listener and explicitly close it. Can you please explain the reason as I might be missing the core issue?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the following comment to holdPort to clarify its use case

// holdPort attempts to bind to the specified server port and holds it until the listener is closed.
// It serves two purposes:
//  1. A successful bind indicates the port is free, meaning the server previously using it is down.
//  2. It prevents other tests from binding to the same port, ensuring this test correctly detects the server as
//     unavailable.

Signed-off-by: Liran Funaro <liran.funaro@gmail.com>
Signed-off-by: Liran Funaro <liran.funaro@gmail.com>

@cendhu cendhu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@cendhu cendhu merged commit c6c7cc0 into hyperledger:main Oct 30, 2025
12 checks passed
@liran-funaro liran-funaro deleted the fix-flaky-test-ports branch October 30, 2025 13:06
Effi-S pushed a commit to Effi-S/fabric-x-committer that referenced this pull request Nov 3, 2025
#### Type of change

- Test update

#### Description

- [ServerConfig] Add retry mechanism when binding to a predefined port
(`c.Listener()`).
- [TestBroadcastDeliver] Use the above retry mechanism to ensure the
port is not used by other tests.

#### Related issues

- resolves hyperledger#109 
- resolves hyperledger#137

---------

Signed-off-by: Liran Funaro <liran.funaro@gmail.com>
Effi-S pushed a commit to Effi-S/fabric-x-committer that referenced this pull request Nov 3, 2025
#### Type of change

- Test update

#### Description

- [ServerConfig] Add retry mechanism when binding to a predefined port
(`c.Listener()`).
- [TestBroadcastDeliver] Use the above retry mechanism to ensure the
port is not used by other tests.

#### Related issues

- resolves hyperledger#109
- resolves hyperledger#137

---------

Signed-off-by: Liran Funaro <liran.funaro@gmail.com>
Signed-off-by: Effi-S <effi.szt@gmail.com>
Effi-S pushed a commit to Effi-S/fabric-x-committer that referenced this pull request Nov 3, 2025
#### Type of change

- Test update

#### Description

- [ServerConfig] Add retry mechanism when binding to a predefined port
(`c.Listener()`).
- [TestBroadcastDeliver] Use the above retry mechanism to ensure the
port is not used by other tests.

#### Related issues

- resolves hyperledger#109
- resolves hyperledger#137

---------

Signed-off-by: Liran Funaro <liran.funaro@gmail.com>
Signed-off-by: Effi-S <effi.szt@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[flaky-test] TestBroadcastDeliver [flaky-test] TestCommitterCMD race condition

2 participants