Merge pull request #241 from telekom/feature/reportng-docs-to-TT2

ReportNG doc update of Testerra 2

Author: martingrossmann

docs/src/docs/execution/test-controller.adoc

@@ -16,6 +16,8 @@ CONTROL.collectAssertions(() -> {
 });
 ----
 
+See <<#_collected_assertions_2, here>> how collected assertions are presented in Report-NG.
+
 == Optional assertions
 
 Optional asserts do not let the test fail, but the assertion message will be added to the log with loglevel `WARN` and will result in an minor failure aspect.
@@ -29,6 +31,8 @@ CONTROL.optionalAssertions(() -> {
 });
 ----
 
+See <<#_optional_assertions_2, here>> how optional assertions are presented in Report-NG.
+
 == Change internal timeout
 
 To change the timeout for internal assertions, you can override it for a specified block.

docs/src/docs/gettingstarted/logging.adoc

@@ -38,13 +38,4 @@ Which results in a log message similar to
 
 The markers `MCID` and `SCID` are referencing to the current MethodContext respectively SessionContext ID.
 
-== Prompt messages
-
-Very important messages can be prompted using:
-
-[source,java]
-----
-log().warn("Important message", Loggable::prompt);
-----
-
-That tells the Report to display these messages in a different way (optional).
+NOTE: Important messages can be prompted to the Report. See <<Priority Messages, here>> for more details.

docs/src/docs/index.adoc

@@ -28,6 +28,7 @@ include::execution.adoc[leveloffset=+1]
 
 '''
 = Testerra Features
+include::reports.adoc[leveloffset=+1]
 include::modules.adoc[leveloffset=+1]
 include::utilities.adoc[leveloffset=+1]
 

docs/src/docs/modules/layout-check.adoc

@@ -22,10 +22,6 @@ The following prerequisites must be met
 . Concrete execution environment: Layout tests should run in a concrete browser environment to ensure the comparability of the screenshots.
 ** Size of browser window: Define fixed size to exclude different sizes of the images at different VM resolutions
 ** Screen resolution and scaling: Make sure you have the same screen resolution and scaling on each presentation device.
-//. Build Server: The library underlying ANNOTATED mode supports the operating systems
-//** Windows 32 - and 64 - Bit
-//** Linux Debian 32 - and 64 - Bit (glibc >= 2.15; Jenkins)
-//** MacOSX (untested)
 
 == Configuration
 
@@ -86,3 +82,7 @@ When you have configured the reference screenshots location and implemented the
 to enable taking automatically screenshots based on the current browser and size configuration and storing them to the reference image's location.
 
 WARNING: All concrete distance values in this {layoutcheck_takereference}-mode will return `0` (zero) and always pass the tests.
+
+== Reporting
+
+Report-NG provides a good presentation of the results of layout checks. See <<#_layout_checks, here>> for more details.

docs/src/docs/reports.adoc

@@ -0,0 +1,13 @@
+= Reporting with Report-NG
+
+include::reports/overview.adoc[leveloffset=+1]
+include::reports/setup.adoc[leveloffset=+1]
+include::reports/dashboard.adoc[leveloffset=+1]
+include::reports/testdetails.adoc[leveloffset=+1]
+include::reports/failure-aspects.adoc[leveloffset=+1]
+include::reports/log-view.adoc[leveloffset=+1]
+include::reports/threads-view.adoc[leveloffset=+1]
+include::reports/teststeps.adoc[leveloffset=+1]
+include::reports/prioritymessages.adoc[leveloffset=+1]
+include::reports/layout-check.adoc[leveloffset=+1]
+include::reports/assertions.adoc[leveloffset=+1]

docs/src/docs/reports/assertions.adoc

@@ -0,0 +1,24 @@
+= More on Assertions
+
+== Collected assertions
+
+When <<#_collected_assertions, Collected assertions>> are used and some of its assertions fail the test is not aborted, but it will be marked as failure in `Report-NG`:
+
+image::report-ng-22.png[align="center", alt="Failed collected assertions"]
+image::report-ng-19.png[align="center", alt="Assert Collector Details"]
+
+The failures are marked in the step view:
+
+image::report-ng-20.png[align="center", alt="Assert Collector Steps"]
+
+
+== Optional assertions
+
+When <<#_optional_assertions, Optional assertions>> fail, the test will also be fully executed and will be marked as passed.
+
+image::report-ng-21.png[align="center", alt="Optional Assert Pass"]
+image::report-ng-17.png[align="center", alt="Optional Assert Details"]
+
+The failed optional asserts will be displayed as warnings:
+
+image::report-ng-18.png[align="center", alt="Optional Assert Steps"]

docs/src/docs/reports/dashboard.adoc

@@ -0,0 +1,22 @@
+= Dashboard View
+
+The dashboard view gives an overview of the last test execution. It shows the total number of executed tests and of passed, skipped or failed tests.
+
+image::report-ng-05.png[align="center", alt="Dashboard"]
+
+On the Breakdown panel is a pie chart that shows the proportions, displayed with different colors according to the test status.
+
+Test status colors:
+[cols="2,^1,4",options="header"]
+|===
+| Color | Test Status | Description
+| Green | Passed | The test was passed.
+| Yellow | Skipped | The test was not executed, because a test it depends on failed.
+| Red | Failed | The test failed.
+| Dark Red | Expected Fail | The test failed, as expected e.g. because of a known bug or problem.
+|===
+
+Check out the <<#Fails, @Fails>> Annotation to see how to mark a test as expected to fail.
+
+Right beside the Breakdown pie chart you find another panel with the Top 3 failure types (aspects).
+There are two more panels below, one displaying detailed information about test execution duration, start and end times and another panel with a list of test classes that reports how many tests did pass, fail or were skipped in the individual test classes.

docs/src/docs/reports/failure-aspects.adoc

@@ -0,0 +1,10 @@
+= Failure Aspects View
+
+In case of many failures it is important to be able to find out the most severe failure reasons (aspects) that caused the most failed tests cases.
+
+For this the Failure Aspect View was designed.
+
+image::report-ng-11.png[align="center", alt="Failure Aspect View"]
+
+It comes with a filter as well and shows the most occurred problems on the top with the number of tests that they caused to fail.
+

docs/src/docs/reports/layout-check.adoc

@@ -0,0 +1,23 @@
+= Layout Checks
+
+In case you are using <<#_layout_check, Layout Checks>> `Report-NG` provides special features to analyze failed layout tests.
+
+== Comparison by Image Size
+
+When a layout test failed because there is a difference of the image size 
+between the actual screenshot and the reference image,
+`Report-NG` will add a failure aspect to the test details panel:
+
+image::report-ng-14.png[align="center", alt="Wrong Image Size Failure Aspect"]
+
+== Comparison by Pixel Distance
+
+When a layout test failed because the percentage of the pixels that are different is too high (pixel distance) `Report-NG` will also add a failure aspect to the test details panel:
+
+image::report-ng-15.png[align="center", alt="High Pixel Distance Size Failure Aspect"]
+
+Furthermore, `Report-NG` provides a special dialog to compare the actual screenshot and the 
+reference image to actually identify the mismatching pixels. It can be opened by clicking one of the three thumbnail images. The pixel comparison dialog provides a diamond shaped slider to change the size of the left and right part of the pixel comparison area. You can also select what is displayed on the left and right part, the actual screenshot, the reference image or the pixel difference:
+
+image::report-ng-16.png[align="center", alt="Pixel Comparison Dialog"]
+

docs/src/docs/reports/log-view.adoc

@@ -0,0 +1,6 @@
+= Log View
+
+The Log View shows all log information that were recorded during test execution.
+It allows to filter by message log level and to search log messages according to a given search term or keyword.
+
+image::report-ng-12.png[align="center", alt="Log View"]