Skip to content

perf(markers): speed up _format_marker and benchmark str(Marker) - #1371

Open
notatallshaw wants to merge 3 commits into
pypa:mainfrom
notatallshaw:perf/format-marker
Open

perf(markers): speed up _format_marker and benchmark str(Marker)#1371
notatallshaw wants to merge 3 commits into
pypa:mainfrom
notatallshaw:perf/format-marker

Conversation

@notatallshaw

@notatallshaw notatallshaw commented Aug 8, 2026

Copy link
Copy Markdown
Member

_format_marker backs str(Marker), and through it Marker.__hash__, Marker.__eq__ and Requirement.__str__.

before after
marker item " ".join([m.serialize() for m in marker]) f"{lhs.serialize()} {op.serialize()} {rhs.serialize()}"
isinstance(marker, list) twice, guard then dispatch once, with the guard folded in
(list, tuple, str), (list, tuple) rebuilt per call module constants
inner (_format_marker(m, first=False) for m in marker) [_format_marker(m, first=False) for m in marker]

A marker item is always (Variable | Value, Op, Variable | Value), so the comprehension and the join are avoidable.

_format_marker takes about 0.75x the time it did, on CPython 3.10, 3.12, 3.14 and 3.15.

TimeMarkerSuite timed the constructor and evaluate but not str, so ASV had no view of this path. time_str adds one.

I started looking at this while reading #1370.

@notatallshaw notatallshaw changed the title Speed up _format_marker perf(markers): speed up _format_marker Aug 8, 2026
@notatallshaw
notatallshaw marked this pull request as draft August 8, 2026 18:09
@notatallshaw notatallshaw changed the title perf(markers): speed up _format_marker perf(markers): speed up _format_marker and benchmark str(Marker) Aug 8, 2026
@notatallshaw

Copy link
Copy Markdown
Member Author

Faster:

| Change   | Before [ef91ddbe]    | After [c4b433f9]    |   Ratio | Benchmark (Parameter)                                                                                  |
|----------|----------------------|---------------------|---------|--------------------------------------------------------------------------------------------------------|
|          | 5.48±0.03ms          | 5.46±0.03ms         |    1    | markers.TimeMarkerSuite.time_constructor [runnervmvrwv9/virtualenv-py3.10-PYTHONHASHSEED0]             |
|          | 4.06±0.03ms          | 4.05±0.02ms         |    1    | markers.TimeMarkerSuite.time_constructor [runnervmvrwv9/virtualenv-py3.11-PYTHONHASHSEED0]             |
|          | 4.66±0.04ms          | 4.69±0.03ms         |    1.01 | markers.TimeMarkerSuite.time_constructor [runnervmvrwv9/virtualenv-py3.12-PYTHONHASHSEED0]             |
|          | 4.69±0.03ms          | 4.69±0.02ms         |    1    | markers.TimeMarkerSuite.time_constructor [runnervmvrwv9/virtualenv-py3.13-PYTHONHASHSEED0]             |
|          | 4.56±0.04ms          | 4.58±0.03ms         |    1.01 | markers.TimeMarkerSuite.time_constructor [runnervmvrwv9/virtualenv-py3.14-PYTHONHASHSEED0]             |
|          | 1.07±0.01ms          | 1.09±0.01ms         |    1.01 | markers.TimeMarkerSuite.time_evaluate [runnervmvrwv9/virtualenv-py3.10-PYTHONHASHSEED0]                |
|          | 749±6μs              | 749±7μs             |    1    | markers.TimeMarkerSuite.time_evaluate [runnervmvrwv9/virtualenv-py3.11-PYTHONHASHSEED0]                |
|          | 879±2μs              | 880±7μs             |    1    | markers.TimeMarkerSuite.time_evaluate [runnervmvrwv9/virtualenv-py3.12-PYTHONHASHSEED0]                |
|          | 826±8μs              | 827±6μs             |    1    | markers.TimeMarkerSuite.time_evaluate [runnervmvrwv9/virtualenv-py3.13-PYTHONHASHSEED0]                |
|          | 783±4μs              | 774±20μs            |    0.99 | markers.TimeMarkerSuite.time_evaluate [runnervmvrwv9/virtualenv-py3.14-PYTHONHASHSEED0]                |
| -        | 542±5μs              | 388±7μs             |    0.72 | markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.10-PYTHONHASHSEED0]                     |
| -        | 373±8μs              | 246±5μs             |    0.66 | markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.11-PYTHONHASHSEED0]                     |
| -        | 407±30μs             | 293±9μs             |    0.72 | markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.12-PYTHONHASHSEED0]                     |
| -        | 392±8μs              | 272±4μs             |    0.69 | markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.13-PYTHONHASHSEED0]                     |
| -        | 401±7μs              | 289±3μs             |    0.72 | markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.14-PYTHONHASHSEED0]                     |

@notatallshaw
notatallshaw marked this pull request as ready for review August 8, 2026 18:20
Comment thread src/packaging/markers.py Outdated
@henryiii

Copy link
Copy Markdown
Contributor

By the way, here or as a followup, we could do #1252 on Marker, which would cut the repeated __hash__ time down. Marker is currently fully immutable. (Specifiers have one mutable property, and Requirements are mutable currently).

Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
@notatallshaw

Copy link
Copy Markdown
Member Author

Specifiers have one mutable property

As a complete aside, I've been thinking of adding a class method calls with_prerelease that allows you to get a new specifier with the prerelease of your choice, and then deprecating the mutability of the prerelease property, not sure what the timeline of that should look like though.

@henryiii

Copy link
Copy Markdown
Contributor

What about supporting __replace__ instead?

@henryiii

Copy link
Copy Markdown
Contributor

All benchmarks:

Change Before [ef91ddb] After [cd3b319] Ratio Benchmark (Parameter)
- 528±5μs 370±5μs 0.7 markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.10-PYTHONHASHSEED0]
- 359±10μs 222±4μs 0.62 markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.11-PYTHONHASHSEED0]
- 384±8μs 252±6μs 0.66 markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.12-PYTHONHASHSEED0]
- 373±7μs 252±4μs 0.68 markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.13-PYTHONHASHSEED0]
- 385±5μs 267±3μs 0.7 markers.TimeMarkerSuite.time_str [runnervmvrwv9/virtualenv-py3.14-PYTHONHASHSEED0]

notatallshaw added a commit to notatallshaw/nab that referenced this pull request Aug 17, 2026
- relation is proposed upstream now; the marker-item
  serialisation is pypa/packaging#1371
- every vendored import uses the from form, so the
  removal plan needs one rule
- prepare_environment and evaluate_prepared are 26.4
notatallshaw added a commit to notatallshaw/nab that referenced this pull request Aug 17, 2026
The private helper names, the SortedOrder alias and release_intervals
all differ from what the fork's drafts carry, and _format_marker is
not pypa/packaging#1371's. Say so, and extend the removal plan to the
vendor job and the references a search-replace cannot reach.
notatallshaw added a commit to notatallshaw/nab that referenced this pull request Aug 20, 2026
- relation is proposed upstream now; the marker-item
  serialisation is pypa/packaging#1371
- every vendored import uses the from form, so the
  removal plan needs one rule
- prepare_environment and evaluate_prepared are 26.4
notatallshaw added a commit to notatallshaw/nab that referenced this pull request Aug 20, 2026
The private helper names, the SortedOrder alias and release_intervals
all differ from what the fork's drafts carry, and _format_marker is
not pypa/packaging#1371's. Say so, and extend the removal plan to the
vendor job and the references a search-replace cannot reach.
notatallshaw added a commit to notatallshaw/nab that referenced this pull request Aug 21, 2026
- relation is proposed upstream now; the marker-item
  serialisation is pypa/packaging#1371
- every vendored import uses the from form, so the
  removal plan needs one rule
- prepare_environment and evaluate_prepared are 26.4
notatallshaw added a commit to notatallshaw/nab that referenced this pull request Aug 28, 2026
- relation is proposed upstream now; the marker-item
  serialisation is pypa/packaging#1371
- every vendored import uses the from form, so the
  removal plan needs one rule
- prepare_environment and evaluate_prepared are 26.4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants