Skip to content

avoid quadratic per-leaf copy in StringMerger.do_transform#5220

Open
sahvx655-wq wants to merge 1 commit into
psf:mainfrom
sahvx655-wq:string-merge-batch-leaves
Open

avoid quadratic per-leaf copy in StringMerger.do_transform#5220
sahvx655-wq wants to merge 1 commit into
psf:mainfrom
sahvx655-wq:string-merge-batch-leaves

Conversation

@sahvx655-wq

Copy link
Copy Markdown
Contributor

Description

Profiling --preview string merging on "%s ..." % (a, b, c, ...) still showed a quadratic after #5199 linearised append_leaves: the run sat around 4s for 8000 operands with len called about 32 million times. StringMerger.do_transform rebuilds the merged line by walking every leaf of the old line and copying each leaf that is not part of a merged string with its own append_leaves(new_line, line, [leaf]) call. The resume hint added in #5199 only persists inside a single append_leaves call, so a one-leaf call always restarts the search for the leaf's position from the front. A % expression's operand tuple is a single parent holding every operand, so copying its n leaves one at a time rescans that child list from index 0 each time and is O(n²). It is reachable pre-auth from blackd with X-Preview and X-Enable-Unstable-Feature: string_processing.

I gather the consecutive leaves that fall outside any merged string group and hand each contiguous run to append_leaves in one call, which is the shape the resume hint was built for, so the copy stays linear. The per-leaf calls were doing nothing the batched call does not, comments included, so the new line is built identically. Output is byte-identical across the test suite and the black source tree in stable, preview and unstable modes plus crafted %-format and implicit-concat inputs; the 8000-operand case drops from roughly 4.1s to 0.8s and the curve flattens from quadratic to linear.

Checklist - did you ...

  • Implement any code style changes under the --preview style, following the stability policy?
  • Add an entry in CHANGES.md if necessary?
  • Add / update tests if necessary?
  • Add new / update outdated documentation?

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

diff-shades results comparing this PR (719b3ee) to main (d7587ce):

--preview style: no changes

--stable style: no changes


What is this? | Workflow run | diff-shades documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant