Skip to content

Clarify Pi0.5 Thor benchmark warmup and latency gate - #180

Open
gcomfident-crypto wants to merge 1 commit into
flashrt-project:mainfrom
gcomfident-crypto:fix/pi05-thor-benchmark-warmup
Open

Clarify Pi0.5 Thor benchmark warmup and latency gate#180
gcomfident-crypto wants to merge 1 commit into
flashrt-project:mainfrom
gcomfident-crypto:fix/pi05-thor-benchmark-warmup

Conversation

@gcomfident-crypto

@gcomfident-crypto gcomfident-crypto commented Aug 19, 2026

Copy link
Copy Markdown

Closes #170

Summary

  • Default the strict Pi0.5 Thor benchmark to 300 warmup calls and record ten ordered latency-group medians.
  • Separate the current README latency references from the older regression baseline without changing the acceptance thresholds.
  • Update the README, benchmark notes, and quickstart help so short Thor warmups are no longer recommended.

The renamed result fields and gate use schema_version: 2, which keeps new artifacts distinct from the old schema.

Testing

  • python -m pytest -q tests/test_pi05_thor_fp4_routing.py (27 passed)
  • python -m pytest -q tests/test_calibration_helpers.py -k stratified_sample_indices (5 passed, 13 deselected)
  • python -m py_compile tests/bench_pi05_decoder_fp4_e2e.py tests/test_pi05_thor_fp4_routing.py examples/quickstart.py
  • python tests/bench_pi05_decoder_fp4_e2e.py --help
  • python examples/quickstart.py --help
  • git diff --check

Thor E2E

I ran the public load_model path at this PR's head, 6f15749f, on NVIDIA Thor with FA4 active. The run used two views, 300 warmups per mode, 100 timed iterations, and the mode-0 clocks locked at GPC 1.575 GHz, NVD 1.692 GHz, and EMC 4.266 GHz.

The historical benchmark fixture is not published. I rebuilt an N=8 fixture from the public LIBERO-10 portion of physical-intelligence/libero at revision a4336d589d589045d1c56423ffdf3b88a0e19b1f. This uses episodes 0 through 378 and the repository's exact stratified_sample_indices(metadata, n=8) output:

[0, 107, 26120, 26248, 26376, 51054, 51189, 75574]

Images came from the raw Parquet PNG bytes and were resized from 256 to 224 with PIL bilinear, with no crop or flip. A second verifier rebuilt every image, wrist image, and state array directly from the Parquets; all eight samples matched byte for byte.

  • Fixture SHA256: aac2bd3629d8cab1e51033ae7ff9fc251ab1d0b0af5ac8ac4283b76034ecae4d
  • Checkpoint SHA256: 9ff277a783d6569989ce0797f8d9539e00b579fd75bbb9a971cf5a1b12cc8f16
  • Norm stats SHA256: b3a44bb2810436fb62917decaea58bd4d9110255df527dea21e8fd40c960bd84
Metric Result Gate
FP8 p50 40.134 ms reference
FP4 p50 27.031 ms at most 34.3 ms
FP4 p95 27.088 ms at most 40 ms
Raw cosine 0.999216 at least 0.995
Worst raw sample 0.997735 at least 0.995
Action cosine 0.999668 at least 0.999
Worst action sample 0.998949 at least 0.995

All seven gates passed. The ten FP4 group medians were:

27.043, 27.038, 27.018, 27.032, 27.027,
27.025, 27.043, 27.023, 27.036, 27.019 ms

The group spread was 0.025 ms, with no late timing step. The formal result.json SHA256 is c043d064392508753854541550d91043f3287a3eefdeee6fb6a817dd48127f17.

I also ran the same benchmark with 20 warmups. It passed and produced byte-identical outputs, but it followed the 300-warmup run and inherited the hot machine state. I am not using that ordering as a cold-start comparison; the cold 20-versus-300 measurements remain the maintainer reproduction in #170.

Runtime versions were Torch 2.11.0a0+a6c236b9fd.nv26.03.46836102, CUDA 13.2, nvidia-cutlass-dsl==4.5.1, and quack-kernels==0.4.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Pi0.5][Jetson Thor] Unable to Reproduce the Documented 27.17 ms 2-View NVFP4 + FA4 Baseline

1 participant