chore(tests): Reduce flakiness due to MockServer websocket errors

[adutra]

Some test flakiness is happening lately, always related to MockServer and how it creates WebSockets.

The flakiness seems to only affect macOS, CI runs fine.

This PR attempts to mitigate the flakiness by tweaking some parameters heuristically:

1. Lower the number of expectations that require WebSockets.
2. Lower the number of test permutations in OAuth2AgentTest.
3. Increase some thread pool sizes for MockServer server and client.
4. Explicitly clear all expectations before closing the server.

[snazy]

LGTM, but mind elaborating on the actual symptoms/reasons what's happening on macOS?

[adutra]

mind elaborating on the actual symptoms/reasons what's happening on macOS?

Unfortunately I didn't keep the stack traces 🤦‍♂️ The most frequent error is thrown from org.mockserver.client.ForwardChainExpectation#registerWebSocketClient.

Let's leave this PR open for a few more days and I'll gather all the stack traces here. There is no rush.

[snazy]

Oh, that's then Netty client related. I suspected the "usual random port stuff", which MockServer clearly implemented properly.

[adutra]

Correct, and I'd also note that this project pushes MockServer to its limits, with a server-per-test pattern and lots of expectations in each test 😅

[adutra]

Another "interesting" observation: I rebooted my laptop yesterday I haven't seen any issues since 🤷‍♂️

[adutra]

There we go again 🙄

First stack trace of the day:

org.mockserver.client.ClientException: Unable to retrieve client registration id
		at org.mockserver.client.ForwardChainExpectation.registerWebSocketClient(ForwardChainExpectation.java:301)
		at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:124)
		at com.dremio.iceberg.authmgr.oauth2.test.expectation.InitialTokenFetchExpectation.create(InitialTokenFetchExpectation.java:24)
		at com.dremio.iceberg.authmgr.oauth2.test.TestEnvironment.createInitialGrantExpectations(TestEnvironment.java:854)
		at com.dremio.iceberg.authmgr.oauth2.test.TestEnvironment.createExpectations(TestEnvironment.java:845)
		at com.dremio.iceberg.authmgr.oauth2.test.TestEnvironment.validate(TestEnvironment.java:104)
		at com.dremio.iceberg.authmgr.oauth2.test.ImmutableTestEnvironment.validate(ImmutableTestEnvironment.java:3052)
		at com.dremio.iceberg.authmgr.oauth2.test.ImmutableTestEnvironment$Builder.build(ImmutableTestEnvironment.java:5368)
		at com.dremio.iceberg.authmgr.oauth2.OAuth2ManagerTest$UnitTests.contextualSessionCacheHit(OAuth2ManagerTest.java:241)
		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
		... 3 more

[snazy]

Another "interesting" observation: I rebooted my laptop yesterday I haven't seen any issues since 🤷‍♂️

"Have you tried to turn it off and on again" worked 🎉

[adutra]

Closing in favor of #205 .