Surface failing CI jobs that only run on main

[pmeier]

On CircleCI we still have these Linux GPU tests:

vision/.circleci/config.yml

Lines 1353 to 1374 in 8499bc4

	- unittest_linux_gpu:
	cu_version: cu117
	name: unittest_linux_gpu_py3.8
	python_version: '3.8'
	- unittest_linux_gpu:
	cu_version: cu117
	filters:
	branches:
	only:
	- main
	- nightly
	name: unittest_linux_gpu_py3.9
	python_version: '3.9'
	- unittest_linux_gpu:
	cu_version: cu117
	filters:
	branches:
	only:
	- main
	- nightly
	name: unittest_linux_gpu_py3.10
	python_version: '3.10'

They are replaced by

vision/.github/workflows/test-linux-gpu.yml

Lines 17 to 21 in 8499bc4

	strategy:
	matrix:
	python_version: ["3.8"]
	cuda_arch_version: ["11.7"]
	fail-fast: false

Although it looks like if this just drops the GPU workflows for Python 3.9 and 3.10, one has to pay attention to the filter key in the CircleCI config: these workflows are only run on the main and the nightly branch.

I can only speak for myself here, but I very rarely have look a the CI on main, but rather operate under the assumption "if the CI in the PR is green, life is good". If the other maintainer feel similar, we are basically running these tests for nothing. Meaning, we can remove them in #7354 without reducing the "apparent coverage".

However, this of course still means that we are reducing coverage. If we can't run this workflows on PRs for monetary reasons, we need a way to surface failing jobs that only run on main. We have such a mechanism already for the dataset availability tests: https://github.com/pytorch/vision/blob/main/.github/workflows/tests-schedule.yml. We just run our tests there on a daily schedule and a bot will open an issue in case something fails like #7361. From there on, we can handle it manually, since this will hopefully not happen often.

cc @seemethere