add an option to show test failures in summary #305
Description
It would be really cool if there was an option to show a list of the test failures again in the summary.
for example in my tests suite a test failure looks like this:
> Task :failgood:test FAILED
SuiteResult > when there are failing contexts ✔ allOk is false
Until ✔ is NotIgnored before now
Until ✔ returns a reason after now
a test context returned by a function ✔ describes behavior
a test context ✔ can tell its name with path
test context defined in a kotlin object ✔ describes behavior
Suite ✔ Empty Suite fails
test context declared on top level ✔ is also a nice way to define your test context
test context defined in a kotlin class ✔ describes behavior
Suite ✔ Suite {} creates a root context
a test context ✔ can be created from a path
a test context ✔ parents is ordered
The describe top level method ✔ creates a context named '<className>' when called with a class
RootContext ✔ knows its file and line number
ObjectContextProvider ✔ provides a context from an class in a kotlin class (MyTest::class.java)
ContextTreeReporter ✔ outputs test results in tree form
ObjectContextProvider ✔ provides a context from an object in a java class (MyTest::class.java)
ContextTreeReporter ✔ outputs empty root context
ObjectContextProvider ✔ provides a context from an object in a kotlin class (MyTest::class)
ContextTreeReporter ✔ outputs empty context
ContextTreeReporter ✔ outputs time
ObjectContextProvider ✔ provides a list of contexts from an object in a kotlin class (MyTest::class)
Error Handling > Non deterministic test names ✔ make their test count as failed
ObjectContextProvider ✔ provides a top level context from a kotlin class
test isolation for root contexts > in the default isolation mode (full isolation) ✔ test dependencies are recreated for each test
SingleTestExecutor > test execution ✔ executes a single test
ObjectContextProvider > correcting source info ✔ corrects the root context source info if its not coming from the loaded class
SingleTestExecutor > test execution ✔ executes a nested single test
SingleTestExecutor ✔ also supports describe / it
test isolation for root contexts > in the default isolation mode (full isolation) ✔ passes testInfo and success to afterEach
SingleTestExecutor > error handling ✔ reports exceptions in the context as test failures
test finder ✔ can find Test classes
ObjectContextProvider > correcting source info ✔ does not touch the source info if it comes from the loaded class
SingleTestExecutor > error handling ✔ reports exceptions in the autoclose lambda as test failures
test isolation for root contexts > a root context with isolation set to false ✔ runs tests without recreating the dependencies
ObjectContextProvider > Error handling ✔ throws when a class contains no contexts
ContextVisitor<*> ✔ can be easily created
test isolation for root contexts > a root context with isolation set to false ✔ passes testInfo and success to afterEach
FailGoodJunitTestEngine ✔ parses a filter string
ObjectContextProvider > Error handling ✔ wraps exceptions that happen at class instantiation: ClassThatThrowsAtCreationTime
ContextDSL::describe ✔ works for class
junit platform observations > UniqueId ✔ replaces + with space when parsing
Error Handling ✔ tests with wrong receiver
ObjectContextProvider > Error handling ✔ wraps exceptions that happen at class instantiation: ClassThatThrowsAtContextGetter
ObjectContextProvider > Error handling ✔ gives a helpful error when a class could not be instantiated
ObjectContextProvider > Error handling ✔ ignores private classes
the ContextFinder ✔ finds a single test with a uniqueId selector
the ContextFinder ✔ does not run a class that has no failgood test annotation
The Junit Platform Engine - works with Blockhound installed
Suite > coroutine scope ✔ does not wait for tests before returning context info
Suite > error handling ✔ treats errors in getContexts as failed context
The Junit Platform Engine ✘ can execute a simple test defined in an object
java.lang.AssertionError: Assertion failed
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invokeSuspend(JunitPlatformFunctionalTest.kt:43)
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invoke(JunitPlatformFunctionalTest.kt)
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invoke(JunitPlatformFunctionalTest.kt)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invokeSuspend(ContextStateCollector.kt:59)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invoke(ContextStateCollector.kt)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invoke(ContextStateCollector.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:100)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:146)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:44)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1.invokeSuspend(ContextStateCollector.kt:56)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
FailGoodTestPluginFactory ✔ provides description and name
createResponse > contexts ✔ creates friendly uniqueid for a root context
FailGoodTestPluginFactory > test framework configuration ✔ returns a suite object that returns no suites
FailGoodTestPluginFactory > test framework configuration ✔ returns the TestUnitFinder
createResponse > contexts ✔ creates friendly uniqueid for a sub context
new mock syntax ✔ looks like this
soft assert syntax ✔ could look like this
Error Handling ✔ handles errors in resource correctly
closing test resources > autoclosable ✔ is closed in reverse order of creation
closing test resources > autoclosable ✔ closes autocloseables without callback
createResponse ✔ creates friendly uuids for tests
createResponse > failed contexts ✔ creates a test node with friendly uniqueid for a failed root context
AssertHelpers > containsExactlyInAnyOrder > simple ✔ returns true when the elements are the same
AssertHelpers > containsExactlyInAnyOrder > simple ✔ returns true false the elements are not the same
AssertHelpers > containsExactlyInAnyOrder > with a class that does not implement comparable ✔ works with list
AssertHelpers > containsExactlyInAnyOrder > with a class that does not implement comparable ✔ works with vararg
AssertHelpers > endsWith ✔ works with list
AssertHelpers > endsWith ✔ works with vararg
another test context defined in a kotlin class ✔ describes behavior
first of multiple contexts defined in one object ✔ describes behavior
examples for the test context dsl ✔ published println calls in the junit runner
retryFor ✔ can call suspend functions
FailGoodJunitTestEngine > can discover tests ✔ returns a root descriptor
FailGoodTestUnitFinder ✔ creates a test unit for each test
closing test resources > autoclosable ✔ works inside a test
closing test resources > autoclosable > error handling > when the test fails and autoclose and aftereach work ✔ calls autoclose callbacks
Root Context Order ✔ is determined by the order field low to high
the ContextFinder > test filtering ✔ supports a classpathRootSelector with a package filter
the ContextFinder > parsing unique id selectors ✔ works when the root contexts contains brackets
a coroutine scope inside failgood tests ✔ works as expected
nice toString functions ✔ ktype has a niceString function
Injecting Test Dependencies ✔ the context can create test dependencies
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ executes all the tests in that context in one go
Given support > error handling ✔ treats errors in the given block as test failures
ExceptionPrettyPrinter ✔ pretty prints the exception with stack trace
Given support ✔ passes the value of the contests given block to the test
ExceptionPrettyPrinter ✔ shortens the stack trace
ExceptionPrettyPrinter ✔ shows only stack trace lines for assertion errors that are in the test file
Junit4Reporter ✔ reports test results
ResourcesCloser > closeAutoCloseables ✔ closes autoclosables
ResourcesCloser > callAfterEach ✔ calls afterEach
ResourcesCloser > callAfterEach > error handling ✔ calls all after each methods even if one fails
ResourcesCloser > callAfterEach > error handling ✔ throws the first exception when multiple afterEach callbacks fail
StringListTestFilter ✔ executes a path that leads to a context
StringListTestFilter ✔ executes a parent of the context
StringListTestFilter ✔ does not execute a different path
StringListTestFilter ✔ executes a root context when the path fits
StringListTestFilter ✔ does not execute a root context when the path is different
TestDSL ✔ publishes a test event for stdout printing
TestDSL ✔ replaces an empty test event to make junit happy
TestDSL ✔ tells the name of the test
ResourcesCloser ✔ handles errors in dependency block
the mocking framework > records function calls ✔ verifies function calls
the mocking framework > records function calls ✔ throws when a verified function was not called
the mocking framework ✔ can return function calls for normal asserting
the mocking framework ✔ has call helpers for up to 5 parameters
the mocking framework ✔ has suspend call helpers for up to 5 parameters
the mocking framework ✔ returns something useful as response to toString
the mocking framework > records function parameters ✔ verifies function parameters
the mocking framework > supports suspend functions ✔ verifies suspend functions
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ does not affect other contexts
the mocking framework > supports suspend functions ✔ throws when parameters don't match
the mocking framework > defining results > with the ✔ defines results via calling the mock
the mocking framework > defining results > with the ✔ defines results via calling the mock even works for nullable functions
the mocking framework > defining results > with the ✔ mocks can throw
the mocking framework > handles equals correctly ✔ returns true for equals with the same mock
the mocking framework > handles equals correctly ✔ returns false for equals with a different object
closing test resources > autoclosable > error handling > when the test fails and autoclose and aftereach work ✔ calls afterEach callbacks
The Junit Platform Engine ✔ can execute a simple test defined in a class
the mocking framework > defining results ✔ can be done when the mock is created
the mocking framework > error handling ✔ detects when the parameter to the is not a mock
second of multiple contexts defined in one object ✔ describes behavior
ContextExecutor > with a valid root context > executing all the tests ✔ returns deferred test results
ContextExecutor > with a valid root context > executing all the tests ✔ returns tests in the same order as they are declared in the file
the mocking framework > parameter placeholders ✔ exist for all basic kotlin types
ContextExecutor > with a valid root context > executing all the tests ✔ returns contexts in the same order as they appear in the file
ContextExecutor ✔ handles failing root contexts
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ calls the given for each test
retryFor ✔ rethrows exception when time is up
sub context isolation > on a root context with default isolation (true) > error handling ✔ does not allow to turn isolation on when it was already off
FailGoodJunitTestEngine > can discover tests ✔ returns all root contexts
retryFor ✔ retries at least once
ContextExecutor > with a valid root context > executing all the tests ✔ reports time of successful tests
ContextExecutor > with a valid root context > executing a subset of tests ✔ does not execute the context at all if the root name does not match
ContextExecutor > with a valid root context > executing all the tests > reports failed tests ✔ reports exception for failed tests
ContextExecutor > with a valid root context > reports line numbers ✔ returns file info for all subcontexts
closing test resources > autoclosable > error handling ✔ reports the test failure even when the close callback fails too
closing test resources > autoclosable > error handling > when the test fails and aftereach fails ✔ calls autoclose callbacks
ContextExecutor > with a valid root context > reports line numbers ✔ returns line number for contexts
closing test resources > autoclosable > error handling > when the test fails and autoclose and aftereach work ✔ reports the test failure
ContextExecutor > with a valid root context > executing a subset of tests ✔ can execute a subset of tests
ContextExecutor > with a valid root context > reports line numbers ✔ reports file name for all tests
ContextExecutor > with a valid root context > reports line numbers ✔ reports line number for all tests
ContextExecutor > supports lazy execution ✔ postpones test execution until the deferred is awaited when lazy is set to true
closing test resources > autoclosable > error handling > when the test succeeds, autoclose fails and aftereach works ✔ calls autoclose callbacks
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ calls callbacks at the correct time
ContextExecutor > timing ✔ reports context structure before tests finish
The Junit Platform Engine ✔ executes contexts that contain tests with the same names
closing test resources > autoclosable > error handling > when the test fails and aftereach fails ✔ calls afterEach callbacks
FailGoodJunitTestEngine > test execution ✔ starts and stops contexts in the correct order
closing test resources > autoclosable > error handling > when the test succeeds, autoclose fails and aftereach works ✔ calls afterEach callbacks
closing test resources > autoclosable > error handling > when the test and autoclose succeeds and aftereach fails ✔ calls autoclose callbacks
a test suite with 1000 tests in one context ✔ runs pretty fast
closing test resources > autoclosable > error handling > when the test and autoclose succeeds and aftereach fails ✔ calls afterEach callbacks
closing test resources > autoclosable > error handling > when the test succeeds, autoclose fails and aftereach works ✔ reports the test failure
The Junit Platform Engine > duplicate test names ✔ are correctly handled in root contexts
closing test resources > after suite callback ✔ is called exactly once at the end of the suite, after all tests are finished
FailGoodJunitTestEngine > test execution ✔ sends one skip event and no start event for skipped tests
retryFor ✔ retries for given time and returns result when it eventually succeeds
closing test resources > autoclosable > error handling > when the test fails and aftereach fails ✔ reports the test failure
The Junit Platform Engine ✔ returns tests in the order that they are declared in the file
closing test resources > autoclosable > error handling > when the test and autoclose succeeds and aftereach fails ✔ reports the test failure
closing test resources > after suite callback ✔ can throw exceptions that are ignored
The Junit Platform Engine > duplicate test names ✔ are correctly handled in sub contexts
The Junit Platform Engine > error handling ✔ correctly reports exceptions in afterEach as test failures
The Junit Platform Engine > error handling ✔ correctly handles test that fail in their second pass
The Junit Platform Engine ✔ returns uniqueIds that it understands (uniqueid round-trip test)
The Junit Platform Engine > duplicate test names ✔ works even in deeply nested contexts
The Junit Platform Engine ✔ works for a failing context or root context
ContextExecutor > when an exception happens inside a subcontext ✔ it is reported as a failing test inside that context
ContextExecutor > when an exception happens inside a subcontext ✔ reports the context as a context
ContextExecutor > an ignored sub-context ✔ it is reported as a skipped test inside that context
ContextExecutor > an ignored sub-context ✔ reports the context as a context
ContextExecutor > detects duplicated tests ✔ fails with duplicate tests in one context
ContextExecutor > detects duplicated tests ✔ does not fail when the tests with the same name are in different contexts
ContextExecutor > detects duplicate contexts ✔ fails with duplicate contexts in one context
ContextExecutor > detects duplicate contexts ✔ does not fail when the contexts with the same name are in different contexts
ContextExecutor > filtering by tag - can filter tests in a subcontext
ContextExecutor > detects duplicate contexts ✔ fails when a context has the same name as a test in the same contexts
ContextExecutor > handles strange contexts correctly ✔ tests can not contain nested contexts
ContextExecutor > filtering by tag ✔ can filter tests in the root context by tag
ContextExecutor > handles strange contexts correctly ✔ a context with only one ignored test
ContextExecutor > filtering by tag ✔ can filter contexts in the root context by tag
182 passing (1.5s)
2 pending
1 failing
185 tests completed, 1 failed, 2 skipped
so I have to scroll up a lot to find the failure that is mentioned in the summary.
it could instead look like this:
> Task :failgood:test FAILED
SuiteResult > when there are failing contexts ✔ allOk is false
Until ✔ is NotIgnored before now
Until ✔ returns a reason after now
a test context returned by a function ✔ describes behavior
a test context ✔ can tell its name with path
test context defined in a kotlin object ✔ describes behavior
Suite ✔ Empty Suite fails
test context declared on top level ✔ is also a nice way to define your test context
test context defined in a kotlin class ✔ describes behavior
Suite ✔ Suite {} creates a root context
a test context ✔ can be created from a path
a test context ✔ parents is ordered
The describe top level method ✔ creates a context named '<className>' when called with a class
RootContext ✔ knows its file and line number
ObjectContextProvider ✔ provides a context from an class in a kotlin class (MyTest::class.java)
ContextTreeReporter ✔ outputs test results in tree form
ObjectContextProvider ✔ provides a context from an object in a java class (MyTest::class.java)
ContextTreeReporter ✔ outputs empty root context
ObjectContextProvider ✔ provides a context from an object in a kotlin class (MyTest::class)
ContextTreeReporter ✔ outputs empty context
ContextTreeReporter ✔ outputs time
ObjectContextProvider ✔ provides a list of contexts from an object in a kotlin class (MyTest::class)
Error Handling > Non deterministic test names ✔ make their test count as failed
ObjectContextProvider ✔ provides a top level context from a kotlin class
test isolation for root contexts > in the default isolation mode (full isolation) ✔ test dependencies are recreated for each test
SingleTestExecutor > test execution ✔ executes a single test
ObjectContextProvider > correcting source info ✔ corrects the root context source info if its not coming from the loaded class
SingleTestExecutor > test execution ✔ executes a nested single test
SingleTestExecutor ✔ also supports describe / it
test isolation for root contexts > in the default isolation mode (full isolation) ✔ passes testInfo and success to afterEach
SingleTestExecutor > error handling ✔ reports exceptions in the context as test failures
test finder ✔ can find Test classes
ObjectContextProvider > correcting source info ✔ does not touch the source info if it comes from the loaded class
SingleTestExecutor > error handling ✔ reports exceptions in the autoclose lambda as test failures
test isolation for root contexts > a root context with isolation set to false ✔ runs tests without recreating the dependencies
ObjectContextProvider > Error handling ✔ throws when a class contains no contexts
ContextVisitor<*> ✔ can be easily created
test isolation for root contexts > a root context with isolation set to false ✔ passes testInfo and success to afterEach
FailGoodJunitTestEngine ✔ parses a filter string
ObjectContextProvider > Error handling ✔ wraps exceptions that happen at class instantiation: ClassThatThrowsAtCreationTime
ContextDSL::describe ✔ works for class
junit platform observations > UniqueId ✔ replaces + with space when parsing
Error Handling ✔ tests with wrong receiver
ObjectContextProvider > Error handling ✔ wraps exceptions that happen at class instantiation: ClassThatThrowsAtContextGetter
ObjectContextProvider > Error handling ✔ gives a helpful error when a class could not be instantiated
ObjectContextProvider > Error handling ✔ ignores private classes
the ContextFinder ✔ finds a single test with a uniqueId selector
the ContextFinder ✔ does not run a class that has no failgood test annotation
The Junit Platform Engine - works with Blockhound installed
Suite > coroutine scope ✔ does not wait for tests before returning context info
Suite > error handling ✔ treats errors in getContexts as failed context
The Junit Platform Engine ✘ can execute a simple test defined in an object
java.lang.AssertionError: Assertion failed
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invokeSuspend(JunitPlatformFunctionalTest.kt:43)
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invoke(JunitPlatformFunctionalTest.kt)
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invoke(JunitPlatformFunctionalTest.kt)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invokeSuspend(ContextStateCollector.kt:59)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invoke(ContextStateCollector.kt)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invoke(ContextStateCollector.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:100)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:146)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:44)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1.invokeSuspend(ContextStateCollector.kt:56)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
FailGoodTestPluginFactory ✔ provides description and name
createResponse > contexts ✔ creates friendly uniqueid for a root context
FailGoodTestPluginFactory > test framework configuration ✔ returns a suite object that returns no suites
FailGoodTestPluginFactory > test framework configuration ✔ returns the TestUnitFinder
createResponse > contexts ✔ creates friendly uniqueid for a sub context
new mock syntax ✔ looks like this
soft assert syntax ✔ could look like this
Error Handling ✔ handles errors in resource correctly
closing test resources > autoclosable ✔ is closed in reverse order of creation
closing test resources > autoclosable ✔ closes autocloseables without callback
createResponse ✔ creates friendly uuids for tests
createResponse > failed contexts ✔ creates a test node with friendly uniqueid for a failed root context
AssertHelpers > containsExactlyInAnyOrder > simple ✔ returns true when the elements are the same
AssertHelpers > containsExactlyInAnyOrder > simple ✔ returns true false the elements are not the same
AssertHelpers > containsExactlyInAnyOrder > with a class that does not implement comparable ✔ works with list
AssertHelpers > containsExactlyInAnyOrder > with a class that does not implement comparable ✔ works with vararg
AssertHelpers > endsWith ✔ works with list
AssertHelpers > endsWith ✔ works with vararg
another test context defined in a kotlin class ✔ describes behavior
first of multiple contexts defined in one object ✔ describes behavior
examples for the test context dsl ✔ published println calls in the junit runner
retryFor ✔ can call suspend functions
FailGoodJunitTestEngine > can discover tests ✔ returns a root descriptor
FailGoodTestUnitFinder ✔ creates a test unit for each test
closing test resources > autoclosable ✔ works inside a test
closing test resources > autoclosable > error handling > when the test fails and autoclose and aftereach work ✔ calls autoclose callbacks
Root Context Order ✔ is determined by the order field low to high
the ContextFinder > test filtering ✔ supports a classpathRootSelector with a package filter
the ContextFinder > parsing unique id selectors ✔ works when the root contexts contains brackets
a coroutine scope inside failgood tests ✔ works as expected
nice toString functions ✔ ktype has a niceString function
Injecting Test Dependencies ✔ the context can create test dependencies
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ executes all the tests in that context in one go
Given support > error handling ✔ treats errors in the given block as test failures
ExceptionPrettyPrinter ✔ pretty prints the exception with stack trace
Given support ✔ passes the value of the contests given block to the test
ExceptionPrettyPrinter ✔ shortens the stack trace
ExceptionPrettyPrinter ✔ shows only stack trace lines for assertion errors that are in the test file
Junit4Reporter ✔ reports test results
ResourcesCloser > closeAutoCloseables ✔ closes autoclosables
ResourcesCloser > callAfterEach ✔ calls afterEach
ResourcesCloser > callAfterEach > error handling ✔ calls all after each methods even if one fails
ResourcesCloser > callAfterEach > error handling ✔ throws the first exception when multiple afterEach callbacks fail
StringListTestFilter ✔ executes a path that leads to a context
StringListTestFilter ✔ executes a parent of the context
StringListTestFilter ✔ does not execute a different path
StringListTestFilter ✔ executes a root context when the path fits
StringListTestFilter ✔ does not execute a root context when the path is different
TestDSL ✔ publishes a test event for stdout printing
TestDSL ✔ replaces an empty test event to make junit happy
TestDSL ✔ tells the name of the test
ResourcesCloser ✔ handles errors in dependency block
the mocking framework > records function calls ✔ verifies function calls
the mocking framework > records function calls ✔ throws when a verified function was not called
the mocking framework ✔ can return function calls for normal asserting
the mocking framework ✔ has call helpers for up to 5 parameters
the mocking framework ✔ has suspend call helpers for up to 5 parameters
the mocking framework ✔ returns something useful as response to toString
the mocking framework > records function parameters ✔ verifies function parameters
the mocking framework > supports suspend functions ✔ verifies suspend functions
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ does not affect other contexts
the mocking framework > supports suspend functions ✔ throws when parameters don't match
the mocking framework > defining results > with the ✔ defines results via calling the mock
the mocking framework > defining results > with the ✔ defines results via calling the mock even works for nullable functions
the mocking framework > defining results > with the ✔ mocks can throw
the mocking framework > handles equals correctly ✔ returns true for equals with the same mock
the mocking framework > handles equals correctly ✔ returns false for equals with a different object
closing test resources > autoclosable > error handling > when the test fails and autoclose and aftereach work ✔ calls afterEach callbacks
The Junit Platform Engine ✔ can execute a simple test defined in a class
the mocking framework > defining results ✔ can be done when the mock is created
the mocking framework > error handling ✔ detects when the parameter to the is not a mock
second of multiple contexts defined in one object ✔ describes behavior
ContextExecutor > with a valid root context > executing all the tests ✔ returns deferred test results
ContextExecutor > with a valid root context > executing all the tests ✔ returns tests in the same order as they are declared in the file
the mocking framework > parameter placeholders ✔ exist for all basic kotlin types
ContextExecutor > with a valid root context > executing all the tests ✔ returns contexts in the same order as they appear in the file
ContextExecutor ✔ handles failing root contexts
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ calls the given for each test
retryFor ✔ rethrows exception when time is up
sub context isolation > on a root context with default isolation (true) > error handling ✔ does not allow to turn isolation on when it was already off
FailGoodJunitTestEngine > can discover tests ✔ returns all root contexts
retryFor ✔ retries at least once
ContextExecutor > with a valid root context > executing all the tests ✔ reports time of successful tests
ContextExecutor > with a valid root context > executing a subset of tests ✔ does not execute the context at all if the root name does not match
ContextExecutor > with a valid root context > executing all the tests > reports failed tests ✔ reports exception for failed tests
ContextExecutor > with a valid root context > reports line numbers ✔ returns file info for all subcontexts
closing test resources > autoclosable > error handling ✔ reports the test failure even when the close callback fails too
closing test resources > autoclosable > error handling > when the test fails and aftereach fails ✔ calls autoclose callbacks
ContextExecutor > with a valid root context > reports line numbers ✔ returns line number for contexts
closing test resources > autoclosable > error handling > when the test fails and autoclose and aftereach work ✔ reports the test failure
ContextExecutor > with a valid root context > executing a subset of tests ✔ can execute a subset of tests
ContextExecutor > with a valid root context > reports line numbers ✔ reports file name for all tests
ContextExecutor > with a valid root context > reports line numbers ✔ reports line number for all tests
ContextExecutor > supports lazy execution ✔ postpones test execution until the deferred is awaited when lazy is set to true
closing test resources > autoclosable > error handling > when the test succeeds, autoclose fails and aftereach works ✔ calls autoclose callbacks
sub context isolation > on a root context with default isolation (true) > turning off isolation for a subcotext ✔ calls callbacks at the correct time
ContextExecutor > timing ✔ reports context structure before tests finish
The Junit Platform Engine ✔ executes contexts that contain tests with the same names
closing test resources > autoclosable > error handling > when the test fails and aftereach fails ✔ calls afterEach callbacks
FailGoodJunitTestEngine > test execution ✔ starts and stops contexts in the correct order
closing test resources > autoclosable > error handling > when the test succeeds, autoclose fails and aftereach works ✔ calls afterEach callbacks
closing test resources > autoclosable > error handling > when the test and autoclose succeeds and aftereach fails ✔ calls autoclose callbacks
a test suite with 1000 tests in one context ✔ runs pretty fast
closing test resources > autoclosable > error handling > when the test and autoclose succeeds and aftereach fails ✔ calls afterEach callbacks
closing test resources > autoclosable > error handling > when the test succeeds, autoclose fails and aftereach works ✔ reports the test failure
The Junit Platform Engine > duplicate test names ✔ are correctly handled in root contexts
closing test resources > after suite callback ✔ is called exactly once at the end of the suite, after all tests are finished
FailGoodJunitTestEngine > test execution ✔ sends one skip event and no start event for skipped tests
retryFor ✔ retries for given time and returns result when it eventually succeeds
closing test resources > autoclosable > error handling > when the test fails and aftereach fails ✔ reports the test failure
The Junit Platform Engine ✔ returns tests in the order that they are declared in the file
closing test resources > autoclosable > error handling > when the test and autoclose succeeds and aftereach fails ✔ reports the test failure
closing test resources > after suite callback ✔ can throw exceptions that are ignored
The Junit Platform Engine > duplicate test names ✔ are correctly handled in sub contexts
The Junit Platform Engine > error handling ✔ correctly reports exceptions in afterEach as test failures
The Junit Platform Engine > error handling ✔ correctly handles test that fail in their second pass
The Junit Platform Engine ✔ returns uniqueIds that it understands (uniqueid round-trip test)
The Junit Platform Engine > duplicate test names ✔ works even in deeply nested contexts
The Junit Platform Engine ✔ works for a failing context or root context
ContextExecutor > when an exception happens inside a subcontext ✔ it is reported as a failing test inside that context
ContextExecutor > when an exception happens inside a subcontext ✔ reports the context as a context
ContextExecutor > an ignored sub-context ✔ it is reported as a skipped test inside that context
ContextExecutor > an ignored sub-context ✔ reports the context as a context
ContextExecutor > detects duplicated tests ✔ fails with duplicate tests in one context
ContextExecutor > detects duplicated tests ✔ does not fail when the tests with the same name are in different contexts
ContextExecutor > detects duplicate contexts ✔ fails with duplicate contexts in one context
ContextExecutor > detects duplicate contexts ✔ does not fail when the contexts with the same name are in different contexts
ContextExecutor > filtering by tag - can filter tests in a subcontext
ContextExecutor > detects duplicate contexts ✔ fails when a context has the same name as a test in the same contexts
ContextExecutor > handles strange contexts correctly ✔ tests can not contain nested contexts
ContextExecutor > filtering by tag ✔ can filter tests in the root context by tag
ContextExecutor > handles strange contexts correctly ✔ a context with only one ignored test
ContextExecutor > filtering by tag ✔ can filter contexts in the root context by tag
182 passing (1.5s)
2 pending
1 failing
failures:
The Junit Platform Engine ✘ can execute a simple test defined in an object
java.lang.AssertionError: Assertion failed
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invokeSuspend(JunitPlatformFunctionalTest.kt:43)
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invoke(JunitPlatformFunctionalTest.kt)
at failgood.junit.it.JunitPlatformFunctionalTest$context$1$1.invoke(JunitPlatformFunctionalTest.kt)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invokeSuspend(ContextStateCollector.kt:59)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invoke(ContextStateCollector.kt)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1$testResult$1.invoke(ContextStateCollector.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:100)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:146)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:44)
at failgood.internal.execution.context.ContextStateCollector$executeTest$1.invokeSuspend(ContextStateCollector.kt:56)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
185 tests completed, 1 failed, 2 skipped