Detect failing standalone tests (fastlane#22273)

* Detect failing standalone tests

* Add the RPSEC_ARGS to allow the user to add more arguments

* Update doc

* Typo

Author: lacostej

Rakefile

@@ -16,8 +16,12 @@ end
 task(:test_all) do
   formatter = "--format progress"
   formatter += " -r rspec_junit_formatter --format RspecJunitFormatter -o #{ENV['CIRCLE_TEST_REPORTS']}/rspec/fastlane-junit-results.xml" if ENV["CIRCLE_TEST_REPORTS"]
-  command = "rspec --pattern ./**/*_spec.rb #{formatter}"
+  command = "rspec --pattern ./**/*_spec.rb #{formatter} #{ENV['RSPEC_ARGS']}"
 
+  run_rspec(command)
+end
+
+def run_rspec(command)
   # To move Ruby 3.0 or next major version migration going forward, we want to keep monitoring deprecation warnings
   if Gem.win_platform?
     # Windows would not work with /bin/bash so skip collecting warnings
@@ -30,6 +34,27 @@ task(:test_all) do
   end
 end
 
+# run, displays and saves the list of tests that do not work standalone
+task(:test_all_individually) do
+  files = Dir.glob("./**/*_spec.rb")
+
+  failed = files.select do |file|
+    formatter = "--format progress"
+    command = "rspec #{formatter} #{ENV['RSPEC_ARGS']} #{file}"
+    run_rspec(command)
+    false
+  rescue => _
+    true
+  end
+
+  unless failed.empty?
+    puts("Individual tests failing: #{failed.join(' ')}")
+    file = "failed_tests"
+    File.write(file, failed.join("\n"))
+    raise "Some tests are failing when ran on their own. See #{file}"
+  end
+end
+
 # Overwrite the default rake task
 # since we use fastlane to deploy fastlane
 task(:push) do

Testing.md

@@ -14,16 +14,24 @@ You can also run those steps independently or on a more fine-grained way.
 
 ### Automated tests
 
-Make sure to run the automated tests using `bundle exec` to ensure you’re running the correct version of `rspec` and `rubocop`
+Make sure to run the automated tests using `bundle exec` to ensure you’re running the correct version of `rspec` and `rubocop`.
 
 #### All unit tests
 
 First, navigate into the root of the _fastlane_ project and run all unit tests using
 
+```
+bundle exec rake test_all
+```
+
+You can also invoke rspec directly
+
 ```
 bundle exec rspec
 ```
 
+The test execution sends all standard output to a random temporary file. Prefix the command line with `DEBUG= ` to print out the output instead. E.g. `DEBUG= bundle exec rspec`
+
 #### Unit tests for one specific tool
 
 If you want to run tests only for one tool, use
@@ -54,6 +62,43 @@ The number is the line number of the unit test (`it ... do`) or unit test group
 
 Instead of using the line number you can also use a filter with the `it "something", now: true` notation and then use `bundle exec rspec -t now` to run this tagged test. (Note that `now` can be any random string of your choice.)
 
+#### Ensuring all tests run independently
+
+If you want to check if all the tests in the test suite can be run independently, use
+
+```
+bundle exec rake test_all_individually
+```
+
+#### Troubleshoot flickering tests
+
+If your tests fail randomly, pass extra arguments to `test_all` and `test_all_individually` using the environment variable `RSPEC_ARGS` to isolate the test failures and reproduce them.
+
+Here are some examples.
+
+Randomize the order of tests for the full suite:
+```
+RSPEC_ARGS="--order rand" bundle exec rake test_all
+```
+
+Pass a given seed to the full test suite:
+```
+RSPEC_ARGS="--seed 1234" bundle exec rake test_all
+```
+
+Run each test file independently and randomize within each run:
+```
+RSPEC_ARGS="--bisect random" bundle exec rake test_all_individually
+```
+
+Run the specific tests in bisect mode with a given seed:
+```
+bundle exec rspec --seed 1234 bisect your/list/of/tests.rb
+```
+
+For more information, see [rspec command line documentation](https://rspec.info/features/3-13/rspec-core/command-line/)
+
+
 ### Code style
 
 To verify and auto-fix the code style