Skip to content

[HUD][disabled tests] Use historical data from json to make chart and metrics #6692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 5, 2025
Merged

[HUD][disabled tests] Use historical data from json to make chart and metrics #6692

merged 1 commit into from
Jun 5, 2025

Conversation

clee2000
Copy link
Contributor

@clee2000 clee2000 commented Jun 2, 2025 •
edited
Loading

Instead of getting info about disabled tests from issues, get it from the table that holds the historical data, populated in #6687, that is created based on the json, or from the json itself

This makes it so we don't need to parse the body for the platforms, and also makes it so that aggregate issues can be handled without more parsing logic

Pros:

  • More closely aligned with what's disabled in CI
  • Take advantage of the bot having already parsed the issues to get all the info so we don't have to reimplement the logic

Cons

  • Not reflective of how many issues are open

Requires the above linked PR to be merged

@vercel Vercel
Copy link

vercel bot commented Jun 2, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
torchci ✅ Ready (Inspect) Visit Preview Jun 3, 2025 11:06pm

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 2, 2025
@clee2000 clee2000 changed the title [disabled tests] Use historical data from json to make chart etc [disabled tests] Use historical data from json to make chart and metrics Jun 2, 2025
@clee2000 clee2000 force-pushed the csl/use_disabled_tests_historical branch from 2c256af to 16d0f11 Compare June 2, 2025 18:13
github-advanced-security[bot]
github-advanced-security bot found potential problems Jun 2, 2025
View reviewed changes
clee2000
clee2000 commented Jun 2, 2025
View reviewed changes
torchci/clickhouse_queries/disabled_test_total/params.json Outdated
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got replaced by fetching disabled-tests-condensed json and getting length from that

@clee2000 clee2000 marked this pull request as ready for review June 2, 2025 18:22
@clee2000 clee2000 requested a review from a team June 2, 2025 18:25
@clee2000 clee2000 changed the title [disabled tests] Use historical data from json to make chart and metrics [HUD][disabled tests] Use historical data from json to make chart and metrics Jun 2, 2025
@huydhn huydhn mentioned this pull request Jun 3, 2025
Merged
@huydhn
Copy link
Contributor

huydhn commented Jun 3, 2025 •
edited
Loading

I think these queries are not working correctly yet, or the table hasn't yet been populated as the number of new disabled tests on https://torchci-git-csl-usedisabledtestshistorical-fbopensource.vercel.app/metrics is still showing up as empty

@clee2000 clee2000 force-pushed the csl/use_disabled_tests_historical branch from b3a3993 to a8b1abd Compare June 3, 2025 23:05
@clee2000
Copy link
Contributor Author

clee2000 commented Jun 3, 2025

I think these queries are not working correctly yet, or the table hasn't yet been populated as the number of new disabled tests on https://torchci-git-csl-usedisabledtestshistorical-fbopensource.vercel.app/metrics is still showing up as empty

The PR it depends on got merged and I backfilled the data so hopefully it should be good now

@huydhn
Copy link
Contributor

huydhn commented Jun 4, 2025 •
edited
Loading

The metric page looks good now. On the other hand, there is a small discrepancy in the preview with 393 disabled tests v.s. prod with 398 of them. The difference looks small enough though. Wdyt?

The number of disabled tests chart on https://torchci-git-csl-usedisabledtestshistorical-fbopensource.vercel.app/kpis looks slightly lower, but its shape remains the same

@clee2000
Copy link
Contributor Author

clee2000 commented Jun 4, 2025

The metric page looks good now. On the other hand, there is a small discrepancy in the preview with 393 disabled tests v.s. prod with 398 of them. The difference looks small enough though. Wdyt?

The number of disabled tests chart on https://torchci-git-csl-usedisabledtestshistorical-fbopensource.vercel.app/kpis looks slightly lower, but its shape remains the same

The five tests here
image
cannot be found in the condensed json because they have unusually names because they are cpp tests. If you go to their issues the verification commit believes they are jobs

@huydhn
Copy link
Contributor

huydhn commented Jun 4, 2025

I see. I guess we would want to close those issues and they are not doing anything, but the bot probably just re-open them again

@clee2000
Copy link
Contributor Author

clee2000 commented Jun 5, 2025

I see. I guess we would want to close those issues and they are not doing anything, but the bot probably just re-open them again

I closed them, they also didn't have recently comments so I don't think the bot will re open them immediately, but it could happen in the future

@clee2000 clee2000 merged commit 576e968 into main Jun 5, 2025
6 checks passed
@clee2000 clee2000 deleted the csl/use_disabled_tests_historical branch June 5, 2025 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@huydhn huydhn huydhn approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@clee2000 @huydhn @facebook-github-bot