Framework-level execution performance: per-step WebDriver round-trips, fixed waits, and serial fixture/parallelism bottlenecks

[lbajsarowicz]

Summary

MFTF is widely perceived as slow. Most discussion blames Selenium/WebDriver, but a framework-level source audit shows that MFTF's own per-step and per-fixture chatter multiplies the base WebDriver round-trip cost, and that several fixed sleeps, redundant waits, and strictly-sequential operations are the larger, more addressable contributors. This issue documents the framework-level (not test-content) bottlenecks with source references against develop (6.0.1, c5149a44), plus a ranked set of optimization areas.

The goal is to open a discussion and land a series of small, independently-verifiable PRs. I'm happy to author them.

Methodology

• Static analysis of src/Magento/FunctionalTestingFramework at develop (6.0.1).
• Market/context research on the Selenium-vs-Playwright speed question and MFTF's current parallelism/reliability posture, with each external claim independently fact-checked. Note: credible, methodology-backed Selenium-vs-Playwright benchmark numbers are scarce — most published figures are vendor blog tables with no hardware/version/methodology. The architectural difference (WebDriver HTTP round-trips vs. persistent CDP/WebSocket) is real, but the takeaways below deliberately target MFTF-side cost, which is fixable today without changing the driver.

---

Root causes

A. Per-step overhead (runs on every test step)

1. A remote getCurrentURL() before every step. Extension/BaseExtension.php:61 calls pageChanged(), which at :96 calls _getCurrentUri() → WebDriver getCurrentURL(). This is a network round-trip on every step, including data/assertion steps that never change the page.

2. A remote browser-log fetch after every step. Extension/TestContextExtension.php:445 calls getLog("browser") on every step. The ENABLE_BROWSER_LOG gate (:448) is only for the Allure attachment; the fetched log is also fed to BrowserLogUtil::logErrors() (:456) which accumulates JS errors for the dontSeeJsError assertion. So the fetch can't simply be removed — but it can be gated behind a "JS error checking enabled" flag and/or fetched once per navigation rather than once per step.

B. Fixed and redundant waits

3. waitForAjaxLoad() always sleeps a fixed 1s. Module/MagentoWebDriver.php:411 calls $this->wait(1) unconditionally after the jQuery-idle check. Because waitForPageLoad() calls waitForAjaxLoad() every time, this is ≥1s of pure sleep per page interaction, across every test. On pages without jQuery the waitForJS at :406 throws and the code falls through to waiting the full pageload_timeout (default 30s via WAIT_TIMEOUT).

4. waitForPageLoad() is three serial waits sharing one large timeout. Module/MagentoWebDriver.php:421 waits for document.readyState, then AJAX idle, then loading masks — each able to consume the full pageload_timeout.

5. Loading-mask wait is chatty. waitForLoadingMaskToDisappear() (:437) loops 8 XPath selectors, calling _findElements() per selector and waitForElementNotVisible() per matched element. This can be a single executeScript() that reports whether any mask is still visible.

6. No pageLoadStrategy configured. etc/config/functional.suite.dist.yml, etc/_envs/chrome.yml, etc/_envs/headless.yml leave the default (normal), so the browser blocks on full load and amOnPage() (:952→:955) immediately re-waits document.readyState. eager + MFTF's explicit waits would remove the redundant block.

7. searchAndMultiSelectOption calls waitForPageLoad() 3× per option (Module/MagentoWebDriver.php:386/388/390), compounding items B3–B6 for every option selected.

C. Session lifecycle

8. Browser session is effectively restarted per test. etc/config/functional.suite.dist.yml:26 sets restart: true. A per-test cookie/storage reset with session reuse (reserving full restart for tagged/dirty tests or retry recovery) avoids paying full browser startup per test.

D. Fixture / data layer (createData/deleteData, magentoCLI)

9. Fixture operations are strictly sequential HTTP, with a new transport constructed per operation — DataGenerator/Persist/CurlHandler.php:143-153 builds an executor per call and :174 closes it. Independent creates/deletes in a test's hooks could run via curl_multi / a keep-alive executor pool.

10. Form-based fixtures log in per executor. DataTransport/AdminFormExecutor.php and FrontendFormExecutor.php authenticate in the constructor (backend GET + login POST + optional 2FA POST) for every entity.

11. REST auth decrypts before checking the token cache. DataTransport/Auth/WebApiAuth.php:63-64 decrypts the admin password before the cache lookup at :82. Reorder to check cache first; cache decrypted credentials per process.

12. magentoCLI is one remote HTTP request per action (Module/MagentoWebDriver.php:540), fetching an admin token each time.

E. Parallelism & reliability

13. No runtime parallelism. MFTF only supports static generation-time sharding (generate:tests --config=parallel → _generated/groups/groupN.txt), balanced by estimated action-weight duration (Util/Sorter/ParallelGroupSorter.php), distributed across CI nodes by the user. Console/RunTestCommand.php:191/231 runs tests and suite groups serially within a node. There is no dynamic work-stealing and no use of historical durations.

14. No automatic retry / flaky-test quarantine. The only recovery is the manual run:failed command. No retry counts, backoff, or quarantine.

15. No changed-module / diff-based test selection. Selection is limited to explicit group/test names and --filter (severity/group). Large suites re-run wholesale.

F. Generation

16. All-or-nothing test loading even for run:test/run:group (Test/Handlers/TestObjectHandler.php extracts/extends every test), a per-node DOMXPath in the XML merge (Config/Dom.php), and all Cest PHP assembled in memory before writing (Util/TestGenerator.php).

---

Ranked recommendations

Tier 1 — small, high-value, low-risk, unit-testable (I plan to PR these first):

#	Change	Root cause
1	Gate the per-step browser-log fetch behind a JS-error-check flag; fetch on navigation rather than every step	A2
2	Remove the unconditional wait(1) in waitForAjaxLoad(); return early when jQuery is absent instead of falling through to the full timeout	B3
3	Collapse waitForLoadingMaskToDisappear() into a single executeScript() poll	B5
4	Avoid the per-step getCurrentURL() on non-navigation steps	A1

Tier 2 — medium effort / config-level:

#	Change	Root cause
5	Default pageLoadStrategy: eager; drop the redundant readyState wait after navigation	B6
6	Split waitForPageLoad() into independently-bounded, shorter sub-timeouts	B4
7	Session reuse with per-test storage reset instead of restart: true	C8
8	Executor/session pool + token/credential caching; curl_multi for independent fixtures	D9–D12

Tier 3 — architectural (need maintainer buy-in; large):

#	Change	Root cause
9	In-node process pool with dynamic work-stealing; historical-duration balancing	E13
10	Built-in retry with quarantine; minimal artifacts on non-final attempts	E14
11	Changed-module / diff-based selective execution	E15
12	Lazy/streaming generation; indexed XML merge	F16

Notes

• Every file/line above is against develop at 6.0.1 (c5149a44) and was re-verified in the checkout, not taken from an older vendored copy.
• Tiers 1–2 are individually measurable with a before/after on a representative run:group. I'll include timing evidence in each PR.
• Happy to split this into separate issues per tier if maintainers prefer.

[m2-assistant]

Hi @lbajsarowicz. Thank you for your report.
To speed up processing of this issue, make sure that you provided sufficient information.
Add a comment to assign the issue: @magento I am working on this

---

Join Magento Community Engineering Slack and ask your questions in #github channel.