Return error when rspec exits with non-zero but no failure

[nprizal]

Description

There’s a scenario where rspec exits with a non-zero code but no test failures or errors are reported. This can occur if there’s an explicit exit call in the source code that isn’t caught within the specs (as shown in the example below). Since no failures are reported, bktec treats it as a “pass”. In this case we should return an error from bktec.

# specs/my_spec.rb
RSpec.describe "test"
  def exits
    exit 6
  end

  it "exits" do
    exits
  end
end

$ rspec specs/my_spec.rb

Finished in 1.26 seconds (files took 6.3 seconds to load)
1 example, 0 failures

$ echo $?
6

[malclocke]

Is there a way we can invert the logic here? We have if ExitError and some inclusion condition and I think this would be a lot better as if ExitError and not some exclusion condition. I.e. change this to "any failure status from the runner is considered an error except for when we explicitly allow it".

This change is good but it leaves us in a similar situation to where we are now, where any unexpected failure state is a pass by default.

[nprizal]

Fair point. But it's more complicated than that. We don't return error from this function when RSpec succesfully run and finished because we want to print the result at the end of the run. Instead, we build a RunResult from the test runner json output, that decided whether bktec will exit with 0 or 1.

I can revert the logic, but we can't simply make "any unexpected failure state is a failed by default", unless we are okay with skipping report printing by default.

[nprizal]

I've updated the logic, let me know what you think.