Wdio cucumber xunit reporter reports assertion errors as errors instead of failures

[christian-bromann]

From @osmolyar on July 20, 2018 17:57

The problem

As seen in the attached xml junit reporter output file, Assertion failures are reported as 'errors' rather than 'failures'. The expected behavior would be for Assertion failures to be reported as failures and other error (navigation, etc.) to be reported as errors.

<?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="license_key1" timestamp="2018-07-19T18:00:20" time="106.514" tests="0" failures="0" errors="0" skipped="0"> <properties> <property name="specId" value="890e23d00e82b9eeee0ec1784156f79e"/> <property name="suiteName" value="License Key1"/> <property name="capabilities" value="chrome"/> </properties> </testsuite> <testsuite name="activate_valid_license_key10" timestamp="2018-07-19T18:00:24" time="19.72" tests="5" failures="0" errors="0" skipped="0"> <properties> <property name="specId" value="890e23d00e82b9eeee0ec1784156f79e"/> <property name="suiteName" value="Activate valid license key10"/> <property name="capabilities" value="chrome"/> </properties> <testcase classname="chrome.Activate valid license key10" name="I log in to the application7" time="102.541"/> <testcase classname="chrome.Activate valid license key10" name="I navigate to the License Key page8" time="92.379"/> <testcase classname="chrome.Activate valid license key10" name="I import and activate the license key /tmp/ISC.cache.key11" time="86.188"/> <testcase classname="chrome.Activate valid license key10" name="The confirmation dialog should contain the expected text13" time="82.835"/> <testcase classname="chrome.Activate valid license key10" name="No error should appear in the response note area14" time="82.831"/> </testsuite> <testsuite name="test_invalid_license_key_files_tmp_expired_cache_key25" timestamp="2018-07-19T18:00:49" time="16.442" tests="4" failures="0" errors="1" skipped="0"> <properties> <property name="specId" value="890e23d00e82b9eeee0ec1784156f79e"/> <property name="suiteName" value="Test invalid license key files /tmp/expired_cache.key25"/> <property name="capabilities" value="chrome"/> </properties> <testcase classname="chrome.Test invalid license key files /tmp/expired_cache.key25" name="I log in to the application7" time="78.443"/> <testcase classname="chrome.Test invalid license key files /tmp/expired_cache.key25" name="I navigate to the License Key page8" time="71.399"/> <testcase classname="chrome.Test invalid license key files /tmp/expired_cache.key25" name="I import and try to activate the license key /tmp/expired_cache.key17" time="65.149"/> <testcase classname="chrome.Test invalid license key files /tmp/expired_cache.key25" name="the response note area should contain ERROR #8604: License key has expired.xx21" time="62.013"> <error message="&#xA;expected 'ERROR #8604: License key has expired.' to include 'ERROR #8604: License key has expired.xx'&#xA;"/> <system-err> <![CDATA[ AssertionError: expected 'ERROR #8604: License key has expired.' to include 'ERROR #8604: License key has expired.xx' at World.<anonymous> (C:/AT-ENSEMBLE/workspace/qd_cucumber_2018/features/step_definitions/LicenseKey.steps.js:37:16) at C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\wdio-sync\build\index.js:594:26 at F (C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\core-js\library\modules\_export.js:35:28) at World.executeSync (C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\wdio-sync\build\index.js:592:12) at C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\wdio-sync\build\index.js:214:29 ]]> </system-err> </testcase> </testsuite> <testsuite name="test_invalid_license_key_files_tmp_invalid_cache_key26" timestamp="2018-07-19T18:01:09" time="16.493" tests="4" failures="0" errors="1" skipped="0"> <properties> <property name="specId" value="890e23d00e82b9eeee0ec1784156f79e"/> <property name="suiteName" value="Test invalid license key files /tmp/invalid_cache.key26"/> <property name="capabilities" value="chrome"/> </properties> <testcase classname="chrome.Test invalid license key files /tmp/invalid_cache.key26" name="I log in to the application7" time="57.91"/> <testcase classname="chrome.Test invalid license key files /tmp/invalid_cache.key26" name="I navigate to the License Key page8" time="50.726"/> <testcase classname="chrome.Test invalid license key files /tmp/invalid_cache.key26" name="I import and try to activate the license key /tmp/invalid_cache.key17" time="44.435"/> <testcase classname="chrome.Test invalid license key files /tmp/invalid_cache.key26" name="the response note area should contain ERROR #8605: License key is invalid.xx21" time="41.426"> <error message="&#xA;expected 'ERROR #8605: License key is invalid.' to include 'ERROR #8605: License key is invalid.xx'&#xA;"/> <system-err> <![CDATA[ AssertionError: expected 'ERROR #8605: License key is invalid.' to include 'ERROR #8605: License key is invalid.xx' at World.<anonymous> (C:/AT-ENSEMBLE/workspace/qd_cucumber_2018/features/step_definitions/LicenseKey.steps.js:37:16) at C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\wdio-sync\build\index.js:594:26 at F (C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\core-js\library\modules\_export.js:35:28) at World.executeSync (C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\wdio-sync\build\index.js:592:12) at C:\Users\quality\AppData\Roaming\npm\node_modules\wdio-cucumber-framework\node_modules\wdio-sync\build\index.js:214:29 ]]> </system-err> </testcase> </testsuite> <testsuite name="test_invalid_license_key_files_tmp_empty_cache_key27" timestamp="2018-07-19T18:01:30" time="16.334" tests="4" failures="0" errors="0" skipped="0"> <properties> <property name="specId" value="890e23d00e82b9eeee0ec1784156f79e"/> <property name="suiteName" value="Test invalid license key files /tmp/empty_cache.key27"/> <property name="capabilities" value="chrome"/> </properties> <testcase classname="chrome.Test invalid license key files /tmp/empty_cache.key27" name="I log in to the application7" time="37.145"/> <testcase classname="chrome.Test invalid license key files /tmp/empty_cache.key27" name="I navigate to the License Key page8" time="29.96"/> <testcase classname="chrome.Test invalid license key files /tmp/empty_cache.key27" name="I import and try to activate the license key /tmp/empty_cache.key17" time="23.765"/> <testcase classname="chrome.Test invalid license key files /tmp/empty_cache.key27" name="the response note area should contain ERROR #8603: Not a valid license key file.21" time="20.819"/> </testsuite> <testsuite name="test_invalid_license_key_files_tmp_cache1_key28" timestamp="2018-07-19T18:01:51" time="16.222" tests="4" failures="0" errors="0" skipped="0"> <properties> <property name="specId" value="890e23d00e82b9eeee0ec1784156f79e"/> <property name="suiteName" value="Test invalid license key files /tmp/cache1.key28"/> <property name="capabilities" value="chrome"/> </properties> <testcase classname="chrome.Test invalid license key files /tmp/cache1.key28" name="I log in to the application7" time="16.497"/> <testcase classname="chrome.Test invalid license key files /tmp/cache1.key28" name="I navigate to the License Key page8" time="9.413"/> <testcase classname="chrome.Test invalid license key files /tmp/cache1.key28" name="I import and try to activate the license key /tmp/cache1.key17" time="3.2"/> <testcase classname="chrome.Test invalid license key files /tmp/cache1.key28" name="the response note area should contain File does not exist21" time="0.287"/> </testsuite> </testsuites>
WDIO.xunitchrome.0-0.zip

Environment

"wdio-cucumber-framework": "^1.1.0",
"wdio-junit-reporter": "^0.3.1",
"wdio-spec-reporter": "0.0.5",
"webdriverio": "^4.6.2",

Node js version: 7.5.0
wdio standalone or grid

Details

Describe in more detail the problem you have been experiencing, if necessary.

Link to Selenium/WebdriverIO logs

Create a gist which is a paste of your full Selenium/WebdriverIO logs, and link them here. Do not paste your full Appium logs here, as it will make this issue very long and hard to read! If you are reporting a bug, always include Selenium/WebdriverIO logs!

Code To Reproduce Issue [ Good To Have ]

Please remember that, with sample code; it's easier to reproduce bug and much faster to fix it.

Copied from original issue: webdriverio/webdriverio#2814