Description
Pitch
In the discussion today on Tree Huggers, @Chillee pointed out that seeing the list of new flaky tests on the PR could help notify developers on potential issue with their change. In that example, HUD page https://hud.pytorch.org/pr/83248 showed flaky tests with the number of retries but it was not immediately visible to developers on the PR body. This lead to a small flood of flaky tests from functorch being disabled after the PR was merged (fixed by pytorch/pytorch#83431)
Ideally, we should show the same information from https://hud.pytorch.org/pr/83248 on the PR body pytorch/pytorch#83248. May be also highlighting new flaky tests to make them stand out.
Solution
We can update Dr. CI message to surface new flaky tests coming from the PR. One aspect that I'm not sure is how effective this would be and how much attention people have on Dr. CI message
cc @janeyx99 @pytorch/pytorch-dev-infra