Fixed Test Order Dependency Flakiness in ExecutableStepsTest.testBoundInputEvent

[LucasEby]

Fixes #1461

Motivation

ExecutableStepsTest.testBoundInputEvent fails when executed on it's own because it is an Order-Dependent test that requires RunEnvironmentSensitiveTest.testRunEnvironmentAllSensitive to initialize shared static state beforehand. Specifically, testRunEnvironmentAllSensitive loads a Spring context that initializes the EncryptionProvider's static AtomicReference<Encryption> cache, which testBoundInputEvent depends on. As a result, the test can fail due to external factors not setting up it's state correctly, despite the test and the code that it is testing remaining unchanged. Order dependent flaky tests can lead to unreliable results from CI and can erode developer trust in the test suite.

Modifications

SlangEntitiesSpringConfig.class was added to the ExecutableStepsTest class' ContextConfiguration, thus providing an Application state context bean to allow the ExecutableStepsTest test suite to run independently of RunEnvironmentSensitiveTest. However, after making this change, ExecutableStepsTest.testBoundInputEvent now polluted RunEnvironmentSensitiveTest.testRunEnvironmentAllSensitive because the EncryptionPRovider class uses a static AtomicReference<Encryption> cache that persists across test classes. When ExecutableStepsTest runs first, it caches its own Encryption bean implementation in this static field, which then gets reused by RunEnvironmentSensitiveTest, causing that test to fail with the wrong encryption behavior.

To solve this order dependency issue, a helper method clearEncryptionProviderCache was introduced which uses reflection to access the private static encryptor field in EncryptionProvider and clears the cached Encryption instance. This method is called before and after each test so that ExecutableStepsTest both starts with a clean state and does not pollute the state for subsequent tests in other test classes.

As a result, these tests are no longer order dependent of one another, as each test class properly initializes and cleans up the shared static encryption provider cache.

Affected Tests

This change affects the existing tests:

• io.cloudslang.lang.runtime.steps.ExecutableStepsTest.testBoundInputEvent
• io.cloudslang.lang.runtime.env.RunEnvironmentSensitiveTest.testRunEnvironmentAllSensitive
  (After adding SlangEntitiesSpringConfig.class to ExecutableStepsTest class' ContextConfiguration, testBoundInputEvent started polluting it's state and causing it to fail. Additional modifications were incorporated to fix this problem)

Tools Used

iDFlakies was utilized to identify the order dependent polluter(s), victim(s), and cleaner(s) in the test suite(s). The tool functions by repeatedly running a project's JUnit tests under many different deterministic test orders to detect flaky tests which pass and fail inconsistently, and then re-executes the failing orders to classify each flaky test as either order-dependent or non-order-dependent. iFixFlakies was executed on the output to further diagnose the source of the problems and additionally generate test helper patches from existing cleaners that were found. With the combination of these tools, I was able to more easily diagnose the source(s) of the problem(s) and create effective solution(s) for them.