Re-run failures only

[gap777]

Is it possible to re-run the tests limited to those that failed on the last execution?

[ilyazub]

@gap777 I use rspec --only-failures for that. Currently, this is not supported with turbo_tests.

I expect that passing the extra options to RSpec will be enough to support this feature (#14 (comment)).

[douglasshuang]

I work with @gap777.

I realized that rspec --only-failures couldn't find failed examples reliably because the parallel runners invoked by turbo_tests were writing to the same status file and clobbering each other's output. I changed our spec/spec_helper.rb to use a different file name per runner:

  config.example_status_persistence_file_path = "spec/examples#{ENV['TEST_ENV_NUMBER']}.txt"

We already use a shell script to invoke turbo_tests, so I added a liine to consolidate those files at the end:

cat spec/examples?*.txt > spec/examples.txt

So for now, we can at least run (non-parallel) rspec --only-failures after turbo_tests.

[ilyazub]

@douglasshuang Thanks to your feedback!

Please note that combining multiple examples*.txt may lead to the RSpec failure. RSpec status persistence files have a header which is expected to be present once (ref: ExampleStatusParser#initialize).

# Concatenate multiple files with the same header
# https://unix.stackexchange.com/a/170692/33308
$ (head -2 spec/examples1.txt && tail -n +3 -q spec/examples?*.txt) > spec/examples.txt

# Checking it worked
$ wc -l spec/examples*.txt
   73015 spec/examples.txt
   46467 spec/examples1.txt
   26550 spec/examples2.txt
  146032 total

# Header is present only once
$ grep example_id spec/examples*.txt
spec/examples.txt:example_id                                                                                                 | status  | run_time               |
spec/examples1.txt:example_id                                                                                                 | status  | run_time               |
spec/examples2.txt:example_id                                                                                                 | status  | run_time               |

[douglasshuang]

@ilyazub Thank you very much for the refinement!

[santigolucass]

@douglasshuang Hey, what was your final solution to run the failed specs in nonparallel?

I would like to talk to you about your parallel process suit. We can do it via email, discord, or whatever you find best

Really appreciate, thanks

[douglasshuang]

@santigolucass Hi, here is the complete solution:

First, we configure RSpec to use a different status persistence file per test runner.

spec/spec_helper.rb:

RSpec.configure do |config|
  ...
  config.example_status_persistence_file_path = "spec/examples#{ENV['TEST_ENV_NUMBER']}.txt"
  ...
end

This produces files named as follows when running turbo_tests:

• spec/examples1.txt
• spec/examples2.txt
• ...

Then, after we run turbo_tests, we combine the files, keeping only one header line.

bin/turbo_tests:

#!/bin/bash

bundle exec turbo_tests $*
(head -n 2 spec/examples1.txt && tail -n +3 -q spec/examples?*.txt) > spec/examples.txt

Since ENV['TEST_ENV_NUMBER'] is not set when running rspec normally, it will use spec/examples.txt:

$ bundle exec rspec --only-failures # This should work as expected now