Parallel test execution#6784
Conversation
sebastianbergmann
added
type/enhancement
API Surface ChangesIf any of the additions below are not intended as public API, mark them with New API SurfaceClasses
Methods
Modified API SurfaceMethods
|
9a08909 to
efcd7ee
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6784 +/- ##
============================================
- Coverage 97.74% 97.59% -0.15%
- Complexity 9303 9749 +446
============================================
Files 905 923 +18
Lines 28530 29872 +1342
============================================
+ Hits 27886 29155 +1269
- Misses 644 717 +73 ☔ View full report in Codecov by Harness. |
sebastianbergmann
changed the title
|
My 2 cents on the topic:
Parallel execution can't and will never be deterministic, a tiny difference between how much a unit lasts results in the following test being run in worker X instead of Y. |
|
Thank you, @Slamdunk, for taking the time to write all of this up. There's a lot of hard-won experience in here, and I appreciate it. I want to set expectations honestly, though: for me, native parallel test execution in PHPUnit is still just an idea, and this pull request is a (remarkably well-working) proof of concept rather than a roadmap commitment. If I ever decide to be serious about this and actually ship it, the feature will have limitations. And those limitations must (and will) be very well documented. Complaints about them will then be kindly refused. 🙂 Even in that case, I neither intend nor expect this to replace dedicated solutions such as ParaTest or Paraunit. The scope I have in mind right now is roughly: PHPUnit supports parallel execution for well-architected test suites that properly deal with resource-usage conflicts like databases, etc. So several of the needs you describe, coordinated bootstrapping, functional/method-level distribution, deterministic replay of a run, are exactly the kind of thing that lives outside that scope and is better served by the tools built specifically for it. That said, your notes are genuinely useful for thinking about where the boundaries of that scope should sit, so thank you again. |
4e03c18 to
5053a2d
Compare
5053a2d to
cf8f973
Compare
|
I have briefly looked into whether the new I/O polling API that PHP 8.6 introduces (RFC: poll_api) could replace the file-based completion polling used by the parallel test runner. The conclusion is that it cannot, because it has the exact limitation that forced the file-based design in the first place: on Windows, it cannot wait on
However:
In other words, Therefore, the uniform file-based completion polling stays. This decision should be revisited only if a future PHP version ships a poll handle that can wait on pipes or process handles on Windows, which is what this feature actually needs. |
cf8f973 to
1e97c6b
Compare
|
I see that it's up to the user to select how many worker to use. The You might be interested in giving it a try, so PHPUnit can provide |
I am aware of |
10747ee to
04cb38d
Compare
Generalize process isolation into a worker that boots PHPUnit once and then runs an arbitrary number of tests, each in response to a command on its control channel. Results are transported back using the same serialized envelope as process isolation, so ChildProcessResultProcessor reconstitutes them unchanged. Tests run by one worker share a process and therefore do not get per-test global-state isolation.
…ially, selected with the new --parallel=<n> CLI option (sequential remains the default). Builds on the persistent worker and async JobRunner from #6753. The distribution unit is one test class, run in a worker that reconstructs and runs it from the same serialized envelope process isolation uses. PHPT tests are distributed too, reconstructed in a worker from their file path. A ResultAggregator replays each unit's collected events into the parent in deterministic suite order, so output, logging, results, and coverage are produced exactly as in sequential mode. Tests that cannot run in a worker are run in the main process at their suite position instead: those marked #[DoNotRunInParallel] (new class- and method-level attribute), those requiring process isolation, those whose data cannot be serialized, and PHPT tests carrying a --DO_NOT_RUN_IN_PARALLEL-- section.
…em inside parallel workers (which hung on Windows)
…instead of multiplexing the workers' output pipes with stream_select(), which hangs on Windows
…FLICTS-- implementation
…r resource partitioning
…test through their IterativeTestSuite when tests are run in parallel
…has finished, instead of each time an outermost test suite finishes
… process so that parallel runs report progress live
…ests are run in parallel, so that only tests of the same suite run concurrently
…ool and the PHPT runner, so that --parallel N never executes more than N units at once
…uites when tests are run in parallel, so that loggers that reconstruct the suite hierarchy work
…estSuite to the worker, so that the suite's event envelope nests them in logger output as in a sequential run
…minating the units that are still executing, so that the --stop-on-* options work with --parallel
…ss, so that a transient crash does not fail a parallel run
…ain process, so that cross-class dependencies work when tests are run in parallel
…s, longest first, so that the longest-running tests do not become the stragglers a parallel run waits for
… after a grace period, so that a parallel run can neither deadlock on an unread pipe nor hang when stopping early
…ected as errored, ship only a unit's own passes in its envelope, and load test-suite bootstrap scripts in workers, so that a parallel run loses no test results and matches a sequential run
…o that the retry of its unit boots a fresh worker process instead of aborting the run or leaking the compromised one
… chunk runs, so that a unit that leads a later chunk is not run inside the test-suite envelope of the chunk before it
…atch with a telling exception when the worker command cannot be encoded, so that a data provider key that is not valid UTF-8 cannot abort a parallel run
…d, and do not start the FILE section of one terminated during SKIPIF, when a parallel run stops early, so that an abandoned test still cleans up after itself
…onflicts with every other test entirely on their own, draining the worker pool and the PHPT runner first, so that the exclusivity these declarations promise actually holds
04cb38d to
43341d8
Compare
…'s classes with named constructors, derivation methods, and shared helpers, so that each invariant lives in one place
…es in one pass through one metadata traversal, and poll a worker's event stream with a stat instead of a re-read, so that a parallel run wastes less work as suites grow
…e helper and two template fragments, writing the configuration and source map once per run for all of them, so that isolated processes and parallel workers cannot drift apart
…e the result envelope's decoding between its two consumers, and let a work unit report its own recorded duration, so that each half of the parallel runner's data model lives in exactly one place
…and the facade's dispatcher selection, and record at collection time which chunk envelopes the units emit, so that behavior the sequential and parallel runners must perform identically is expressed exactly once
2 participants
PHPUnit can now execute a test suite across several worker processes concurrently instead of one test after another. Parallel execution is opt-in via a new
--parallel=<n>command-line option and changes nothing when it is not used: the sequentialTextUI\TestRunnerremains the default and parallel mode is selected only when<n> > 1.Native parallelism here is a generalization of process isolation rather than a new subsystem: a worker reconstructs and runs a unit of work and ships its outcome home in the very same serialized envelope that process isolation already uses, and the parent replays that envelope through its normal event pipeline. As a result the parent process remains the single source of truth for all output, logging, results, and code coverage, which are produced exactly as in sequential mode.
How it works
#[BeforeClass]/#[AfterClass]and intra-class ordering.Runner\Parallel\WorkerPoolownsnpersistent workers and multiplexes their control channels withstream_select(), pulling from a dynamic work-stealing queue so the load self-balances against stragglers.Runner\Parallel\ResultAggregatorbuffers each unit and releases it only once every preceding unit (in suite order) has been released, so the event stream, and therefore every output format, including the default progress output, is byte-for-byte what sequential mode produces.Running in the main process
Some tests cannot, or must not, run in a worker. These are run in the main process at their correct suite position (the aggregator runs them in place while releasing results, so global ordering is preserved):
#[DoNotRunInParallel]— a new attribute, valid on classes and methods, for tests that must not run alongside others (for instance because they share a process-global resource).#[RunInSeparateProcess],#[RunTestsInSeparateProcesses], or global--process-isolation) — a shared worker cannot provide isolation, but the main process spawns the isolated child as usual.serialize()throw) as well as resources (whichserialize()silently degrades to0).--CONFLICTS--section — the PHPT equivalent of the attribute, since a PHPT file cannot carry PHP attributes.New public API
--parallel=<n>CLI option (also listed in--help).#[PHPUnit\Framework\Attributes\DoNotRunInParallel](TARGET_CLASS | TARGET_METHOD), with full metadata-layer support.--CONFLICTS--section as supported by the PHP project's PHPT runner.PHPUNIT_WORKER_ID— the small, stable ordinal (0,1,2, ...), ideal for indexing a fixed set of pre-provisioned resources.PHPUNIT_WORKER_TOKEN— a value of the form<id>_<random>that is unique across workers and across runs, for resources that must not collide with those left behind by a previous run.PHPUnit's own test suite
Running PHPUnit's own end-to-end suite in parallel surfaced a number of PHPT tests that rely on a fixed shared path in
sys_get_temp_dir()(a sharedphpunit.xml, the result cache, atest-resultsfile) and therefore race when run concurrently. These 36 tests — inmigration/,cli/validate-configuration/, and the result-cache/duration/defects-ordering tests inexecution-order/— have been marked with--DO_NOT_RUN_IN_PARALLEL--so they run in the main process.Results
--testsuite unit --parallel 10produces the same results as sequential mode (same test and assertion counts), stable across runs. Note that which worker runs which test class is not deterministic — the work-stealing scheduler assigns classes by timing — but the aggregated output is identical regardless.--testsuite end-to-end --parallel 10: 1105 tests, 0 failures, stable across runs; wall-clock dropped from ~3m18s (everything in the main process) to ~34s on a multi-core machine.Notes and limitations (possible follow-ups)
#[DoNotRunInParallel].#[Depends]co-location,--stop-on-*early halt of in-flight workers, and retrying a unit on a fresh worker after a crash are not yet implemented.