Skip to content

avoid quadratic tree traversal in pre_order/post_order/leaves#5235

Open
sahvx655-wq wants to merge 2 commits into
psf:mainfrom
sahvx655-wq:iterative-tree-traversal
Open

avoid quadratic tree traversal in pre_order/post_order/leaves#5235
sahvx655-wq wants to merge 2 commits into
psf:mainfrom
sahvx655-wq:iterative-tree-traversal

Conversation

@sahvx655-wq

Copy link
Copy Markdown
Contributor

Description

Profiling blackd on a deeply nested expression such as a long a ** b ** c ** ... power chain showed pre_order and leaves in blib2to3/pytree.py dominating the trace and growing quadratically. get_features_used runs one pre_order walk over the whole tree on every format pass, so it is an easy place to measure: that walk alone took 8.8ms at 500 terms, 33.9ms at 1000, 132ms at 2000 and 520ms at 4000, quadrupling every time the depth doubled. All three traversal helpers (pre_order, post_order, leaves) descend with yield from child.<method>(), and because yield from resumes every delegating generator in the chain on each step, one full walk costs O(depth) per node and is O(n^2) once the tree is n levels deep. This pass runs before any formatting, so a deeply nested submission is a cheap pre-auth amplification vector through unauthenticated blackd.

Replacing the recursive delegation with an explicit stack visits each node once regardless of depth. The traversal order is unchanged: I checked it leaf-for-leaf and in pre/post-order against the recursive form on assorted trees, and formatting is byte-identical across the whole tests/data corpus in stable, preview and unstable modes plus the full test suite (434 passed). Keeping the fix in the traversal helpers themselves linearises every caller at once (get_features_used, normalize_fmt_off, the --line-ranges walk and the rest) instead of each having to special-case deep input; the same 4000-term walk drops from ~520ms to ~5.9ms and the curve goes from quadratic to linear.

Checklist - did you ...

  • Implement any code style changes under the --preview style, following the stability policy?
  • Add an entry in CHANGES.md if necessary?
  • Add / update tests if necessary?
  • Add new / update outdated documentation?

@sahvx655-wq
sahvx655-wq force-pushed the iterative-tree-traversal branch from dae9801 to ca203a5 Compare July 13, 2026 16:34
@github-actions

Copy link
Copy Markdown
Contributor

diff-shades results comparing this PR (faaeb51) to main (51abf53):

--preview style: no changes

--stable style: no changes


What is this? | Workflow run | diff-shades documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant