Fixed Bean and Encryption Issues Causing Order-Dependent Flakiness in Multiple Tests

[LucasEby]

Fixes #1465

Motivation

The following (victim) tests exhibit order-dependent behavior and fail when EncryptorConfigTest.testEncryptionWithoutEncryptorImplementation (the polluter) runs before them, unless intermediate tests clean the shared static state:

• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValue
• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted
• io.cloudslang.fortest.SensitiveValueTest.testSensitiveValueEncryptDecrypt
• io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive

When EncryptorConfigTest.testEncryptionWithoutEncryptorImplementation runs first it removes the encryption bean from the application context, leaves EncryptionProvider.encryptor in a polluted state, and may leave SlangEntitiesSpringConfig.applicationContext pointing to the wrong context. The victim tests attempt to create SensitiveValue instances which require encryption but they fail because they either use a cached encryptor with the wrong context, the encryption bean cannot be resolved because the application context is misconfigured, or "Application context bean is missing" errors are triggered.

Additionally, DeserializeTest.testDeserializeInput fails unless SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted (or another test that properly initializes the state) is run beforehand. This is because the test requires the SlangEntitiesSpringConfig.applicationContext to be set to a context containing a DummyEncryptor bean and the EncryptionProvider.encryptor cache to be cleared or properly initialized.

Order dependent flaky tests can lead to unreliable results from CI and can erode developer trust in the test suite.

Modifications

A new abstract base class SpringEncryptionTestBase was introduced to ensure static state is properly initialized before each test method. It clears the EncryptionProvider.encryptor cache using reflection to reset the static AtomicReference and injects the test's ApplicationContext into SlangEntitiesSpringConfig.applicationContext to ensure the correct Spring context is used. Each of the affected test classes (ValueFactoryTest, SensitiveValueTest, and DeserializeTest) now extend this new abstract base class.

Additionally, a DummyEncryptor bean configuration was added to DeserializeTest to ensure it has the required encryption implementation in its Spring context.

Affected Tests

This change affects the existing tests:

• io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive
• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValue
• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted
• io.cloudslang.fortest.SensitiveValueTest.testSensitiveValueEncryptDecrypt
• io.cloudslang.lang.entities.DeserializeTest.testDeserializeInput

Tools Used

iDFlakies was utilized to identify the order dependent polluter(s), victim(s), and cleaner(s) in the test suite(s). The tool functions by repeatedly running a project's JUnit tests under many different deterministic test orders to detect flaky tests which pass and fail inconsistently, and then re-executes the failing orders to classify each flaky test as either order-dependent or non-order-dependent. iFixFlakies was executed on the output to further diagnose the source of the problems and additionally generate test helper patches from existing cleaners that were found. With the combination of these tools, I was able to more easily diagnose the source(s) of the problem(s) and create effective solution(s) for them.

[LucasEby]

I had to check out an older version of the repo to utilize the above mentioned tools since they do not support the current JDK version. After opening the PR I discovered that some of the tests had already been fixed so the proposed parent class is no longer needed.

[LucasEby]

This PR was re-opened after investigating the previously accepted PR 1427 and determining that it did not entirely fix the order dependent issues that were identified in the SensitiveValueTest's methods. While the previous solution correctly identified one of the polluters and mitigated it, it did not mitigate all of the order dependent sources of pollution which cause the SensitiveValueTest's methods to fail. This is because there are other polluters that exist in the test suite such as ValueFactoryTest.testCreatePyObjectValueFromValueSensitive which still pollute the tests, causing them to fail.