Skip to content

Multiple Order Dependent Flaky Test Failures Due to Missing Application Context Beans and Uncleared Encryptor Cache #1465

@LucasEby

Description

@LucasEby

The following (victim) tests exhibit order-dependent behavior and fail when EncryptorConfigTest.testEncryptionWithoutEncryptorImplementation (the polluter) runs before them, unless intermediate tests clean the shared static state:

  • io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValue
  • io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted
  • io.cloudslang.fortest.SensitiveValueTest.testSensitiveValueEncryptDecrypt
  • io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive
    
When EncryptorConfigTest.testEncryptionWithoutEncryptorImplementation runs first it removes the encryption bean from the application context, leaves EncryptionProvider.encryptor in a polluted state, and may leave SlangEntitiesSpringConfig.applicationContext pointing to the wrong context. The victim tests attempt to create SensitiveValue instances which require encryption but they fail because they either use a cached encryptor with the wrong context, the encryption bean cannot be resolved because the application context is misconfigured, or "Application context bean is missing" errors are triggered.

Additionally, DeserializeTest.testDeserializeInput fails unless SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted (or another test that properly initializes the state) is run beforehand. This is because the test requires the SlangEntitiesSpringConfig.applicationContext to be set to a context containing a DummyEncryptor bean and the EncryptionProvider.encryptor cache to be cleared or properly initialized.

Order dependent flaky tests can lead to unreliable results from CI and can erode developer trust in the test suite.

I have included the easily reproducible failures below:

Test 1 Failures:
mvn -pl cloudslang-entities -Dtest=io.cloudslang.lang.entities.DeserializeTest#testDeserializeInput test -e

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.cloudslang.lang.entities.DeserializeTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.604 s <<< FAILURE! - in io.cloudslang.lang.entities.DeserializeTest
[ERROR] io.cloudslang.lang.entities.DeserializeTest.testDeserializeInput Time elapsed: 0.183 s <<< ERROR!
java.lang.RuntimeException: Application context bean is missing.
at io.cloudslang.lang.entities.DeserializeTest.testDeserializeInput(DeserializeTest.java:56)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] DeserializeTest.testDeserializeInput:56 » Runtime Application context bean is ...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

After fixing the Application context bean issue, the test still fails because no encryptors are found:

mvn -pl cloudslang-entities -Dtest=io.cloudslang.lang.entities.DeserializeTest#testDeserializeInput test -e

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.cloudslang.lang.entities.DeserializeTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.566 s <<< FAILURE! - in io.cloudslang.lang.entities.DeserializeTest
[ERROR] io.cloudslang.lang.entities.DeserializeTest.testDeserializeInput Time elapsed: 0.168 s <<< ERROR!
java.lang.RuntimeException: No encryptors found
at io.cloudslang.lang.entities.DeserializeTest.testDeserializeInput(DeserializeTest.java:70)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] DeserializeTest.testDeserializeInput:70 » Runtime No encryptors found
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

mvn -pl cloudslang-entities -Dtest=io.cloudslang.lang.entities.bindings.values.ValueFactoryTest#testCreatePyObjectValueFromValueSensitive test -e

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.cloudslang.lang.entities.bindings.values.ValueFactoryTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.341 s <<< FAILURE! - in io.cloudslang.lang.entities.bindings.values.ValueFactoryTest
[ERROR] io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive Time elapsed: 0.018 s <<< ERROR!
java.lang.RuntimeException: Application context bean is missing.
at io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive(ValueFactoryTest.java:40)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ValueFactoryTest.testCreatePyObjectValueFromValueSensitive:40 » Runtime Applic...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

Test 2 Failures:
mvn -pl cloudslang-entities -Dtest=io.cloudslang.lang.entities.bindings.values.ValueFactoryTest#testCreatePyObjectValueFromValueSensitive test -e

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.cloudslang.lang.entities.bindings.values.ValueFactoryTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.341 s <<< FAILURE! - in io.cloudslang.lang.entities.bindings.values.ValueFactoryTest
[ERROR] io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive Time elapsed: 0.018 s <<< ERROR!
java.lang.RuntimeException: Application context bean is missing.
at io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive(ValueFactoryTest.java:40)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ValueFactoryTest.testCreatePyObjectValueFromValueSensitive:40 » Runtime Applic...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions