process.exit() is not setting "1" upon a failed test scenario

[Golshy]

Summary ✍️

My team is having an issue where we have a .feature that includes multiple scenarios. If one of those scenarios fails, our CI pipeline in Jenkins is still displaying a SUCCESS and 🟢 highlight on the dashboard

We are using the Cucumber reporter and find that after it has finished running, when calling echo $? we receive a 0

This is not the same situation as in NW v1.7 which returns a code of 1 and passes the FAILURE through our CI pipelines

Can anyone please advise us on how to troubleshoot this or if to log an issue for this problem?

[swrdfish]

Is this intermittent or does this happen everytime?

[vaibhavsingh97]

Also, please provide which cucumber version you are using with v2?

[Golshy]

Thank you @swrdfish and @vaibhavsingh97 for your quick responses.

More information

• We are using NightWatch v2 (2.2.3)
• We are using NightWatch-API (3.0.2) in conjunction with NWv2
• We are using @cucumber/cucumber (7.3.2)

Response to questions

1. This is a consistent issue that happens every time there is any failed test scenario included in a feature file run
2. @cucumber/cucumber (7.3.2) is used

[vaibhavsingh97]

Nightwatch API is not compatible with Nightwatch v2. We have in-house cucumber integration. Please refer to: https://nightwatchjs.org/guide/writing-tests/using-cucumberjs.html#using-cucumberjs-with-nightwatch

[kcng4506]

@vaibhavsingh97 We found that nightwatch-api works fine with no modifications to its code, as all the required legacy commands are still present in Nightwatch v2.
We have legacy code that does not work with the in-house Nightwatch cucumber integration:
We load the webdriver once and the home page once per a given feature (i.e. test suite). Each Scenario then proceeds from the last page from the previous Scenario. In the in-house Nightwatch cucumber integration, the webdriver is loaded and unloaded for each Scenario. We need to be able to open a web site and proceed from there for subsequent Scenarios.

We tried the alternative solution here https://github.com/nightwatchjs-community/cucumber-nightwatch, because we make use of tags and other properties of the cucumber command line. (cucumber cli options --tags are fixed in #2978). We needed modifications and refactoring of our scripts and it does not handle screenshots on failure.
We also need the screenshot feature when an assertion fails or a waitForXXX command fails (because our automation runs are headless) in each Scenario. This does not happen with this alternative solution, so for the time being we are using nightwatch-api.

[Golshy]

Follow up 🔀

Extra Testing
We also ran a test scenario using NWv2 with no NightWatch-API enabled and encountered the same problem. If there is a failure in the test scenarios, after the script completes, cucumber is not passing an exit code 1.

Best Guess 🧠
My team suspects that process-listnr.js is overriding the exit code and enforcing it to always have 0

Workaround Found for CI ☑️
We found a solution for our CI by running shell script within our Jenkins groovy configuration:

if [ ! -e report/cucumber_report.html ]; then exit 1; fi
                                
echo about to run grep
if [ `grep -c "Scenarios Failed" reports/cucumber_report.html` = 0 ]; then echo "script passed"; 
else echo "script failed"; 
fi
if [ `grep -c "Scenarios Failed" reports/cucumber_report.html` = 0 ]; then exit 0; 
else exit 1; 
fi

Additionally we are handling if our cucumber_report.html file does not exist as a result of some unexpected failure during script runs ⭐

Conclusion 🗒️
We are now able to capture the appropriate exit code for a full passed workflow and show 🟢 on the Jenkins Dashboard. And, for a workflow containing any failures, we capture exit code 1 and pass it to get a FAILURE and 🔴 : on the Jenkins Dashboard.

The only open question we have now is whether this is a legitimate defect and if we should log it on the Issues board in NightWatch Github?

CC: @vaibhavsingh97 @swrdfish @beatfactor

[vaibhavsingh97]

I tried reproducing the issue, but I was not able to. Can you please share a sample repo/code which helps me reproducing the issue

[Golshy]

Hi @vaibhavsingh97

You can refer this this sample code and have it run a failed test step : #3221

Feature

@Sample
Scenario: Nested WaitForElementVisible Callbacks
Given I am on the website
Then Working sample waitForElementVisible
Then Not working sample waitForElementVisible

Step-Defs

Then(/^Not working sample waitForElementVisible$/, async () => {
                await client.waitForElementVisible(ref.home_click.selector, 3000, true, async function (result){
                    if(result.status == 0){
                        await myFunction(ref.signin_link.selector);
                    }
                });
                await client.click(ref.home_click.selector);

                return client;
            });

            Then(/^Working sample waitForElementVisible$/, async () => {
                await myFunction(ref.signin_link.selector);
                await client.click(ref.home_click.selector);

                return client;
            });

            async function myFunction (element){
                await client
                .waitForElementVisible(element, 3000, false, function (result) {
                    console.log("********result ************", result.status, result.value)
                })
            }

The same result happens and cucumber does not return with exit code 1 as expected, instead it returns 0