Clarify Pi0.5 Thor benchmark warmup and latency gate - #180
Open
gcomfident-crypto wants to merge 1 commit into
Open
Clarify Pi0.5 Thor benchmark warmup and latency gate#180gcomfident-crypto wants to merge 1 commit into
gcomfident-crypto wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #170
Summary
The renamed result fields and gate use
schema_version: 2, which keeps new artifacts distinct from the old schema.Testing
python -m pytest -q tests/test_pi05_thor_fp4_routing.py(27 passed)python -m pytest -q tests/test_calibration_helpers.py -k stratified_sample_indices(5 passed, 13 deselected)python -m py_compile tests/bench_pi05_decoder_fp4_e2e.py tests/test_pi05_thor_fp4_routing.py examples/quickstart.pypython tests/bench_pi05_decoder_fp4_e2e.py --helppython examples/quickstart.py --helpgit diff --checkThor E2E
I ran the public
load_modelpath at this PR's head,6f15749f, on NVIDIA Thor with FA4 active. The run used two views, 300 warmups per mode, 100 timed iterations, and the mode-0 clocks locked at GPC 1.575 GHz, NVD 1.692 GHz, and EMC 4.266 GHz.The historical benchmark fixture is not published. I rebuilt an N=8 fixture from the public LIBERO-10 portion of
physical-intelligence/liberoat revisiona4336d589d589045d1c56423ffdf3b88a0e19b1f. This uses episodes 0 through 378 and the repository's exactstratified_sample_indices(metadata, n=8)output:Images came from the raw Parquet PNG bytes and were resized from 256 to 224 with PIL bilinear, with no crop or flip. A second verifier rebuilt every image, wrist image, and state array directly from the Parquets; all eight samples matched byte for byte.
aac2bd3629d8cab1e51033ae7ff9fc251ab1d0b0af5ac8ac4283b76034ecae4d9ff277a783d6569989ce0797f8d9539e00b579fd75bbb9a971cf5a1b12cc8f16b3a44bb2810436fb62917decaea58bd4d9110255df527dea21e8fd40c960bd84All seven gates passed. The ten FP4 group medians were:
The group spread was 0.025 ms, with no late timing step. The formal
result.jsonSHA256 isc043d064392508753854541550d91043f3287a3eefdeee6fb6a817dd48127f17.I also ran the same benchmark with 20 warmups. It passed and produced byte-identical outputs, but it followed the 300-warmup run and inherited the hot machine state. I am not using that ordering as a cold-start comparison; the cold 20-versus-300 measurements remain the maintainer reproduction in #170.
Runtime versions were Torch
2.11.0a0+a6c236b9fd.nv26.03.46836102, CUDA 13.2,nvidia-cutlass-dsl==4.5.1, andquack-kernels==0.4.1.