Skip to content

[CRCR] Add failed-tests-detail input to callback action and ingest path - #8607

Draft
subinz1 wants to merge 1 commit into
mainfrom
crcr-failed-tests-ingest
Draft

[CRCR] Add failed-tests-detail input to callback action and ingest path#8607
subinz1 wants to merge 1 commit into
mainfrom
crcr-failed-tests-ingest

Conversation

@subinz1

@subinz1 subinz1 commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds support for per-test failure details in the CRCR pipeline, addressing the ingest portion of #8545.

  • Callback action: new failed-tests-detail input accepting a JSON array of {name, classname, message, duration} objects
  • report_callback.py: passes the new field as workflow.failed_tests_detail, capped at 50 entries
  • crcrUtils.ts: extracts and persists as failed_tests_json in DynamoDB/ClickHouse, with message truncation at 500 chars

Size caps

Cap Value Rationale
Max entries 50 Prevents DynamoDB 400KB item overflow
Max message length 500 chars Enough for assertion line, not full stacktrace

Usage (downstream workflow)

- uses: pytorch/test-infra/.github/actions/cross-repo-ci-relay-callback@main
  with:
    status: completed
    conclusion: failure
    failed-tests-detail: |
      [
        {"name": "test_foo", "classname": "TestBar", "message": "AssertionError: expected True"},
        {"name": "test_baz", "classname": "TestQux", "message": "RuntimeError: CUDA OOM"}
      ]

Test plan

  • Verify callback action passes the field through to the relay
  • Verify extractDynamoRecord persists failed_tests_json when present
  • Verify size caps work (>50 entries truncated, long messages clipped)
  • Verify missing/empty failed-tests-detail doesn't break existing flow

Closes the ingest path portion of #8545. UI drill-down will follow in a separate PR.

@vercel

vercel Bot commented Aug 24, 2026

Copy link
Copy Markdown

Deployment failed for project torchci with the following error:

You don't have permission to create a Preview Deployment for this Vercel project: torchci.

View Documentation: https://vercel.com/docs/accounts/team-members-and-roles

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 24, 2026
@subinz1
subinz1 marked this pull request as draft August 24, 2026 12:12
@subinz1

subinz1 commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

Addresses #8545 (ingest path portion).

Add support for per-test failure details in the CRCR pipeline. The
callback action now accepts a `failed-tests-detail` JSON array input
containing test name, classname, message, and duration for each failed
test. The ingest path (crcrUtils.ts) extracts and persists it as
`failed_tests_json` in DynamoDB/ClickHouse.

Size caps applied at both layers: max 50 entries, message truncated
to 500 chars. This prevents unbounded growth in DynamoDB items while
giving backend maintainers enough detail to identify which tests broke.

Addresses #8545 (ingest path portion).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CRCR] Failed-test drill-down from RFC-0054 is absent end-to-end (failed_tests_json is never populated)

1 participant