Run.Parallel can return fewer files than it was given, and report the run as successful.
Seen on PS7 - macOS 14 in this run. Four files went in, three came back. The test that caught it, preserves discovery order of containers, only caught it because it asserts on the container names:
Expected: 'A.Tests.ps1,B.Tests.ps1,C.Tests.ps1,D.Tests.ps1'
Actual : 'A.Tests.ps1,C.Tests.ps1,D.Tests.ps1'
B.Tests.ps1 was not reported as failed, skipped or errored. It was not reported at all. Without an assertion on the names the run would have looked green with 3 tests instead of 4.
The worker that had B.Tests.ps1 died while importing Pester:
Collection was modified; enumeration operation may not execute.
The expression after '&' in a pipeline element produced an object that was not valid.
It must result in a command name, a script block, or a CommandInfo object.
The 'Invoke-Pester' command was found in the module 'Pester', but the module could not
be loaded due to the following error: [Should operator 'Be' is not registered]
There are two separate problems here.
The run does not notice a worker that returned nothing
Invoke-TestInParallel filters the worker results in src/functions/Pester.Parallel.ps1:
# Keep only well-formed worker results (defensive against stray pipeline output).
$results = @($results | & $SafeCommands['Where-Object'] { $_ -is [System.Management.Automation.PSCustomObject] -and $null -ne $_.PSObject.Properties['Containers'] })That line cannot tell stray pipeline output from a worker that died before it returned its result object, and nothing afterwards compares $results.Count to $work.Count. So a lost file is dropped as quietly as a stray string, and Sort-Object then puts the survivors in discovery order, which is why the output above looks like a normal three file run.
This part does not depend on the import race. Any worker that dies for any reason costs a whole test file and says nothing.
#3004 stopped one failing worker from aborting the results of the others. It did not add a check that every file came back.
The import race itself
Concurrent Import-Module Pester from several worker runspaces. #2901 fixed one instance of this by serializing Pester's own writes to PowerShell's process global Verbs.s_validVerbs. Collection was modified is thrown by an enumerator rather than by an insert, so this is a different one, and the lock in VerbsPatcher does not cover a reader that is inside PowerShell's own code and never takes that lock. AllowShouldVerb also schedules a Remove for five seconds after every import, so there is a structural modification of that shared dictionary long after import returns.
I could not reproduce it locally on macOS, PowerShell 7.5.5:
- 288 concurrent
Import-Module calls with start times staggered across the five second window, clean.
- 200 real
Run.Parallel runs over 8 files, 1600 worker imports, no short run.
- 40 bursts of 10 imports timed to land on the five second removal, 400 imports, clean.
So the mechanism above is a hypothesis, not a conclusion. The CI log is the only evidence for it so far.
🤖
Run.Parallelcan return fewer files than it was given, and report the run as successful.Seen on
PS7 - macOS 14in this run. Four files went in, three came back. The test that caught it,preserves discovery order of containers, only caught it because it asserts on the container names:B.Tests.ps1was not reported as failed, skipped or errored. It was not reported at all. Without an assertion on the names the run would have looked green with 3 tests instead of 4.The worker that had
B.Tests.ps1died while importing Pester:There are two separate problems here.
The run does not notice a worker that returned nothing
Invoke-TestInParallelfilters the worker results insrc/functions/Pester.Parallel.ps1:That line cannot tell stray pipeline output from a worker that died before it returned its result object, and nothing afterwards compares
$results.Countto$work.Count. So a lost file is dropped as quietly as a stray string, andSort-Objectthen puts the survivors in discovery order, which is why the output above looks like a normal three file run.This part does not depend on the import race. Any worker that dies for any reason costs a whole test file and says nothing.
#3004 stopped one failing worker from aborting the results of the others. It did not add a check that every file came back.
The import race itself
Concurrent
Import-Module Pesterfrom several worker runspaces. #2901 fixed one instance of this by serializing Pester's own writes to PowerShell's process globalVerbs.s_validVerbs.Collection was modifiedis thrown by an enumerator rather than by an insert, so this is a different one, and the lock inVerbsPatcherdoes not cover a reader that is inside PowerShell's own code and never takes that lock.AllowShouldVerbalso schedules aRemovefor five seconds after every import, so there is a structural modification of that shared dictionary long after import returns.I could not reproduce it locally on macOS, PowerShell 7.5.5:
Import-Modulecalls with start times staggered across the five second window, clean.Run.Parallelruns over 8 files, 1600 worker imports, no short run.So the mechanism above is a hypothesis, not a conclusion. The CI log is the only evidence for it so far.
🤖