Skip to content

perf(buckets): add compilation buckets below 256 tokens - #706

Open
teerthsharma wants to merge 1 commit into
google-deepmind:mainfrom
teerthsharma:perf/add-small-compilation-buckets
Open

perf(buckets): add compilation buckets below 256 tokens#706
teerthsharma wants to merge 1 commit into
google-deepmind:mainfrom
teerthsharma:perf/add-small-compilation-buckets

Conversation

@teerthsharma

@teerthsharma teerthsharma commented Jul 30, 2026

Copy link
Copy Markdown

Summary

_BUCKETS starts at 256, so every input below 256 tokens is padded up to 256 before the trunk runs. This adds 64, 96, 128, 192, 384. All thirteen existing sizes are retained, so nothing is padded more than today.

No change to model code, config, outputs, or --buckets semantics.

Padding cost

tokens bucket before → after padded cost before → after
64 256 → 64 5.97x → 1.00x
100 256 → 128 3.52x → 1.36x
128 256 → 128 2.59x → 1.00x
300 512 → 384 2.44x → 1.48x
384 512 → 384 1.64x → 1.00x

Ratios use a whole-model operation count derived from the op shapes in modules.py, evoformer.py, diffusion_head.py, confidence_head.py:
F(N) = 976896*N^3 + 1433624448*N^2 + 585859918336*N.
Worst case over 64..5120: 5.97x → 2.15x. Mean, log-uniform: 1.45x → 1.32x.

Validation

Against the committed branch:

  • Default parsed back from source is strictly increasing, as pipeline.py:53 requires; all 13 previous sizes present.
  • Of 5,120 token counts in 1..5120: 0 padded more than before, 320 padded less, rest unchanged.
  • python -m py_compile run_alphafold.py passes.

Limits

Operation counts, not wall-clock; at small N inference is closer to launch-bound, so the realised gain is smaller. Not exercised end to end — the model parameters are access-gated and were unavailable, so the checks above are static ones on the bucket list and padding arithmetic. Benefit is confined to inputs below ~400 tokens. Each added bucket costs one compilation the first time an input falls into it; workloads that never run small inputs are unaffected.

AI disclosure

Per CONTRIBUTING.md: AI (Claude) was used for the analysis and for this description. Topology and geometric methods are my area of specialisation, and the padding arithmetic and bucket reasoning here sit in that area. The operation counts are reproducible from the formula and bucket list above. No inference run was performed, so nothing here is a manually tested result.

@google-cla

google-cla Bot commented Jul 30, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@teerthsharma
teerthsharma force-pushed the perf/add-small-compilation-buckets branch from e3211ef to 47b2820 Compare July 30, 2026 01:22
The default bucket list starts at 256, so every input with fewer tokens is
padded up to 256. Trunk cost grows faster than linearly in the padded token
count, so a 100-token input does several times the work it needs to.

Adds 64, 96, 128, 192 and 384. All thirteen existing bucket sizes are retained,
so no input is padded more than it is today. A bucket only triggers a
compilation when an input actually falls into it, so the added sizes cost
nothing for workloads that never use them.

Signed-off-by: teerthsharma <teerths57@gmail.com>
@teerthsharma
teerthsharma force-pushed the perf/add-small-compilation-buckets branch from 47b2820 to 1db8499 Compare July 30, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant