Flaky TestFailFast tests: Result did not match the golden fixture

[Ikke]

Description

At least on certain architectures, we notice some of the TestFailFast/Task/task and TestFailFast/Option/default tests are flaky.

On riscv64:

    executor_test.go:1082: Result did not match the golden fixture. Diff is below:
        
        --- Expected
        +++ Actual
        @@ -1,2 +1,2 @@
        +dep2
         
        -

Different build (3.46.4):

  executor_test.go:1061: Result did not match the golden fixture. Diff is below:
        
        --- Expected
        +++ Actual
        @@ -1,2 +1,2 @@
        +dep1
         
        -
  executor_test.go:1076: Result did not match the golden fixture. Diff is below:
        
        --- Expected
        +++ Actual
        @@ -1,2 +1,2 @@
        +dep1
         
        -

On ppc64le:

   executor_test.go:1097: Result did not match the golden fixture. Diff is below:
        
        --- Expected
        +++ Actual
        @@ -1,2 +1,3 @@
        +dep2
        +dep3
         
        -

After retrying a couple of times, these tests eventually pass.

Version

3.47.0

Operating system

Alpine Linux

Experiments Enabled

No response

Example Taskfile

# N/A

[trim21]

also on Windows

[trulede]

@Ikke @trim21 you both might like to try and run the individual flaky tests and see if you can quantify how often they are failing. It does not take long to get a feeling and can be helpful.

More interesting in this case is why the golden fixtures seem to have no output, but somehow there is output? Is that correct? These are dependency tests, as they run in parallel, getting deterministic output is difficult. That could also be a factor.

[Ikke]

Still doing further testing, but it appears these tests are affected by other tests. Running them in isolation (with `-run), I haven't been able to reproduce the test failure.

[trulede]

@Ikke I was wondering if that might be the case. Which run command are you using (or which tests are failing)?

[Ikke]

These pass:

• go test -v -count 10 . -run TestFailfast/Task/task
• go test -v -count 10 . -run TestFailfast/Default/option
• go test -v -count 10 . -run TestFailfast

This fails frequently:

• go test -v -count 10 .
  • TestFailfast/Task/task 8 out of 10
  • TestFailfast/Option/default 5 out of 10

[Ikke]

This is on riscv64, which is not the fastest hardware, so that may influence the timing of things.

[trulede]

@Ikke I think in file testdata/failfast/task/Taskfile ... which looks flakey ... push the sleep values to 10,11,12 respectively and see if that helps. In that test you want the 4th dep to exit while the other 3 are sleeping, and I guess for failfast to be useful, the sleeping deps would be signalled.

The problem with thest failfast/Option is that it uses the default taskfile and adds the option failfast, so that default taskfile becomes flakey as a result. In that case I would add a new taskfile specifically of the failfast/Option, copy from default, but push the sleep values as per the previous case.

[Ikke]

Applying this patch seems to indeed solve / reduce the test failures:

diff --git a/testdata/failfast/default/Taskfile.yaml b/testdata/failfast/default/Taskfile.yaml
index 079ad05..c382123 100644
--- a/testdata/failfast/default/Taskfile.yaml
+++ b/testdata/failfast/default/Taskfile.yaml
@@ -8,7 +8,7 @@ tasks:
       - dep3
       - dep4

-  dep1: sleep 0.1 && echo 'dep1'
-  dep2: sleep 0.2 && echo 'dep2'
-  dep3: sleep 0.3 && echo 'dep3'
+  dep1: sleep 11 && echo 'dep1'
+  dep2: sleep 12 && echo 'dep2'
+  dep3: sleep 13 && echo 'dep3'
   dep4: exit 1
diff --git a/testdata/failfast/task/Taskfile.yaml b/testdata/failfast/task/Taskfile.yaml
index 078ed17..518085e 100644
--- a/testdata/failfast/task/Taskfile.yaml
+++ b/testdata/failfast/task/Taskfile.yaml
@@ -9,7 +9,7 @@ tasks:
       - dep4
     failfast: true

-  dep1: sleep 0.1 && echo 'dep1'
-  dep2: sleep 0.2 && echo 'dep2'
-  dep3: sleep 0.3 && echo 'dep3'
+  dep1: sleep 10 && echo 'dep1'
+  dep2: sleep 11 && echo 'dep2'
+  dep3: sleep 12 && echo 'dep3'
   dep4: exit 1

Still I feel the test should be improved, I even saw it fail once on a fast x86_64 server.