Description
TestNG Version
6.12, 6.11, ...
Expected behavior
According to official documentation parameter 'alwaysRun' for before-methods is used only to enable/disable filtering by groups. It must not force method execution if higher-level configuration(s) failed.
In any case alwaysRun should not be used both for filtering and for execution forcing
Actual behavior
BeforeTest/BeforeClass/BeforeMethod with (alwaysRun = true) is executed when higher - level configuration failed (BeforeSuite/BeforeTest/BeforeClass)
Is the issue reproductible on runner?
- Shell
- Maven
- Gradle
- Ant
- Eclipse
- IntelliJ
- NetBeans
Test case sample
Please, share the test case (as small as possible) which shows the issue
public class AlwaysRunTests {
@BeforeSuite
public void failedBeforeSuite() {
System.out.println("I'm beforeSuite. I should be alone in console log. Going to fail...");
throw new RuntimeException();
}
@BeforeTest(alwaysRun = true)
public void beforeTest() {
System.out.println("BUG: I'm beforeTest and I was invoked (((( ");
throw new RuntimeException();
}
@BeforeClass(alwaysRun = true)
public void beforeClass() {
System.out.println("BUG: I'm beforeClass and I was invoked (((( ");
throw new RuntimeException();
}
@BeforeMethod(alwaysRun = true)
public void beforeMethod() {
System.out.println("BUG: I'm beforeMethod and I was invoked (((( ");
throw new RuntimeException();
}
@Test
public void testMethod() {
System.out.println("I'm testMethod");
}
}Console output (stacktraces truncated):
I'm beforeSuite. I should be alone in console log. Going to fail...
java.lang.RuntimeException
at qa.ok.testng.AlwaysRunTests.failedBeforeSuite(AlwaysRunTests.java:14)
BUG: I'm beforeTest and I was invoked ((((
java.lang.RuntimeException
at qa.ok.testng.AlwaysRunTests.beforeTest(AlwaysRunTests.java:21)
BUG: I'm beforeClass and I was invoked ((((
java.lang.RuntimeException
at qa.ok.testng.AlwaysRunTests.beforeClass(AlwaysRunTests.java:28)
BUG: I'm beforeMethod and I was invoked ((((
java.lang.RuntimeException
at qa.ok.testng.AlwaysRunTests.beforeMethod(AlwaysRunTests.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:599)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:877)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1201)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:776)
at org.testng.TestRunner.run(TestRunner.java:634)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:425)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:420)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:385)
at org.testng.SuiteRunner.run(SuiteRunner.java:334)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1318)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1243)
at org.testng.TestNG.runSuites(TestNG.java:1161)
at org.testng.TestNG.run(TestNG.java:1129)
at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
Test ignored.
===============================================
Default Suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 4, Skips: 0
===============================================