Bug: Does not select an affected JUnit test suite annotated with @RunWith(Suite.class)

[Chenguang-Zhu]

Summary: If one uses @RunWith(Suite.class), dependencies do not
include code under test (or test classes that are in the suite). This
is a safety issue, i.e., STARTS may not select a test with changed
behavior.

Steps to Reproduce: Unzip the attached file. Run the bash
script ./s provided in this directory. This script first runs test with
STARTS, then change the return value of a method under test,
finally runs the same test with STARTS again.

Expected Results: p.RegressionTest runs in both the first and the
second execution. It passes in the first execuion, and it fails in
the second execution.

Actual Results: p.RegressionTest runs only in the first execution.
The test is not selected in the second execution.

starts_bug_test_suite.zip

[owolabileg]

@Chenguang-Zhu, thanks for reporting! Preliminary investigation shows that the problem is not about @RunWith(Suite.class), but rather, ClassTest0.class is not a compile-time dependency of p.RegressionTest. Here's the jdeps output for p.RegressionTest:

$ jdeps -v -cp ~/.m2/repository/junit/junit/4.12/junit-4.12.jar ./target/test-classes/p/RegressionTest.class
RegressionTest.class -> /home/owolabi/.m2/repository/junit/junit/4.12/junit-4.12.jar
RegressionTest.class -> /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar
   p.RegressionTest                                   -> java.lang.Object
   p.RegressionTest                                   -> org.junit.runner.RunWith                           junit-4.12.jar
   p.RegressionTest                                   -> org.junit.runners.Suite                            junit-4.12.jar
   p.RegressionTest                                   -> org.junit.runners.Suite$SuiteClasses               junit-4.12.jar

Even if you run jdeps recursively (using the command below), it simply does not find ClassTest0.class to be a dependency of p.RegressionTest:

$ jdeps -v -R -cp ~/.m2/repository/junit/junit/4.12/junit-4.12.jar ./target/test-classes/p/RegressionTest.class