[float8nocompile] Temporarily disable float8nocompile CI tests due to flakiness on sm89

[danielvegamyhre]

float8 does not work properly on sm89, the tests only pass on H100s.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1902

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 7196d56 with merge base c376285 ():

NEW FAILURE - The following job has failed:

• .github/workflows/float8nocompile_test.yaml (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[msaroufim]

You will almost never have signal on main in a reasonable timeframe if you merge this (16h as of this comment). There's hardly ever any H100 available in pool.

[jainapurva]

You will almost never have signal on main in a reasonable timeframe if you merge this (16h as of this comment). There's hardly ever any H100 available in pool.

Agreed, we can't run a lot of tests on H100. This test is flaky on SM-89, either we can disable the flaky tests on SM89 or run it on H100 with reduced frequency (nightly or only on commit in main)

[danielvegamyhre]

Ok let's disable this test for now and I'll take a look at why it's flaky on sm89

[danielvegamyhre]

@jainapurva this is ready for another look when you have a chance. I just commented out the jobs: ... for now.

[jainapurva]

Ok let's disable this test for now and I'll take a look at why it's flaky on sm89

This looks good.