Skip to content

ci: cover protected runner queue time - #1065

Closed
chaofengw-nv wants to merge 1 commit into
NVIDIA:mainfrom
chaofengw-nv:ci/extend-internal-wait-budget
Closed

ci: cover protected runner queue time#1065
chaofengw-nv wants to merge 1 commit into
NVIDIA:mainfrom
chaofengw-nv:ci/extend-internal-wait-budget

Conversation

@chaofengw-nv

Copy link
Copy Markdown
Collaborator

Background

The first post-merge live validation on the dedicated external-contributor PR #1056 exposed a real runner-queue timeout. Public bridge run 33080320518 correctly correlated private run 33080349314 by nonce, but its Wait for the exact Internal CI result step failed after 56m53s while the selected fast_foundation_stereo GPU job was still queued with no steps started. The failure was therefore in the bridge waiting budget, not in nonce correlation, artifact generation, or failure-log rendering.

Exit Criteria

  • The public bridge wait budget covers protected runner queueing plus the longest supported Internal model proof.
  • The bridge retains time after the private run completes to download, validate, print, upload, and publish the failure result.
  • The exact-run nonce correlation and fail-closed behavior remain unchanged.
  • A deterministic workflow test locks down the timeout and polling budget.

Implementation

  • Raises the trusted dispatch job timeout from 60 to 360 minutes.
  • Raises the exact private-run polling window from 220 to 1,400 15-second polls, or 350 minutes.
  • Reserves the final 10 minutes for artifact transfer, closed-contract validation, public .log publication, and status update.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Validation

Commands and Results

  • Regression test before the workflow change: failed because the dispatch timeout was 60 rather than 360 minutes.
  • Focused bridge test after the change: 1 passed.
  • python3 -m tools.community_ci source-quality --base github/main: passed; 158 tests passed, changed-file lint/formatting passed, and the complexity gate passed.
  • git diff --check: passed.

Hardware, Environment, and Revisions

  • Source head: 870bf4087921acaac9c8f4a8edf138fa6093b8b8.
  • Source base: 7aa0b211e52ad17f6d827d4c33022b36130e7f01.
  • Live evidence: public bridge 33080320518, private Internal run 33080349314, Source PR test(stereo): cover DHWC8 and probe GPU CI logs #1056 head d3b8f6650f110773a6a54c6d37fcf00203bffb80.
  • The private GPU job remained queued for more than 55 minutes with no steps, reproducing the budget failure without executing contributor code in the public workflow.

Not Run / Remaining Gaps

The complete public failure-log handoff could not run because the original bridge exhausted its 55-minute polling window before a protected GPU runner became available. After this change reaches main, rerun only #1056 and verify the nonce-correlated artifact, printed .log, status target, root CTest failure block, and sensitive-data checks end to end.

Notes For Future Readers

The longest Internal model proof currently has a 180-minute job timeout. The 350-minute polling window covers that execution budget plus substantial protected-runner queueing while staying within GitHub's six-hour hosted-job limit. If Internal model timeouts grow beyond 180 minutes, revisit both budgets together.

Risk level

  • Low
  • Medium
  • High

Risk rationale: this changes only how long the trusted bridge waits for an already exact, nonce-correlated private run. Permissions, payload contents, publication checks, and status context are unchanged.

Signed-off-by: chaofengw <chaofengw@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b8ea547a-b0e7-4f1e-b961-428f7832d1e7

📥 Commits

Reviewing files that changed from the base of the PR and between 7aa0b21 and 870bf40.

📒 Files selected for processing (2)
  • .github/workflows/internal-ci-bridge.yml
  • tests/tools/test_github_actions_ci.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes

    • Increased the CI bridge timeout to support longer-running validation jobs.
    • Extended polling time so builds have more time to complete before being marked as failed.
  • Tests

    • Updated automated checks to verify the extended timeout and polling limits.

Walkthrough

The internal CI bridge now supports a 360-minute dispatch timeout and up to 1,400 polling iterations. Tests verify both workflow limits.

Changes

Internal CI bridge timing

Layer / File(s) Summary
Extend CI timing and validate limits
.github/workflows/internal-ci-bridge.yml, tests/tools/test_github_actions_ci.py
The dispatch timeout increased to 360 minutes. The exact Internal CI run polling limit increased to 1,400 iterations. Tests assert both values.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 870bf

This PR extends the trusted CI bridge wait budget and adds deterministic coverage without changing permissions, payloads, or publication behavior; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: yifeif-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. (1 skipped: 1 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description check ✅ Passed The description covers the background, exit criteria, implementation, change category, validation results, revisions, remaining gaps, risk, and rollout context. It clearly identifies the end-to-end va…
Title check ✅ Passed The title is concise and accurately identifies the primary change: extending CI coverage for protected runner queue time.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description covers the background, exit criteria, implementation, change category, validation results, revisions, remaining gaps, risk, and rollout context. It clearly identifies the end-to-end validation that remains after merge.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. (1 skipped: 1 unsupported.)


Comment @coderabbitai help to get the list of available commands.

@chaofengw-nv chaofengw-nv added the run-internal-ci Maintainer-approved dispatch to internal CI label Aug 27, 2026
@github-actions github-actions Bot removed the run-internal-ci Maintainer-approved dispatch to internal CI label Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant