Do not run CI with Julia pre

[devmotion]

Same as and with the same motivation as JuliaStats/StatsBase.jl#992.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.71%. Comparing base (63373be) to head (5792870).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #194   +/-   ##
=======================================
  Coverage   69.71%   69.71%           
=======================================
  Files          16       16           
  Lines         799      799           
=======================================
  Hits          557      557           
  Misses        242      242           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andreasnoack]

I think it can be useful and not too noisy to test on prereleases. Would it be possible to run JET only on release?

[devmotion]

No, not if we want to keep using the standard Pkg test environments (in extras or test/Project
toml). CI errors on pre due to the Pkg resolver not being able to install the test dependencies. Basically, currently CI on pre only works if pre is the latest stable release (ie when there is currently no prerelease and pre is identical to 1).

[andreasnoack]

Should we consider running JET tests in a separate environment?

[devmotion]

I considered it but I thought it would not be worth it.

It would require activating a separate environment in the tests, which in my experience in eg OrdinaryDiffEq did not work 100% reliably, as the Pkg resolver is invoked twice and packages are only carried over implicitly from the base test environment. Probably there's some tweaks one could apply to make it more reliable than my previous experience in OrdinaryDiffEq and eg MCMCChains, but it seemed not worth it and much easier and more reliable to keep working with a single environment.

A second point is that "pre" does not necessarily imply that tests are run with a pre-release. Whenever the latest release is a stable release and not a pre-release, then the CI run with "pre" is identical to the one with "1".

[andreasnoack]

My main concern is that get failing CI on master after a Julia minor release. If JET isn't updated for prereleases then it might not be updated in time for the actual release. Having failing CI on master is quite bad, so if JET can't be used on prereleases then I'm not sure if it is worth running as part of CI.

[devmotion]

I thought a bit more about it, and I think the right approach would be to fix compat bounds in the registry and skip JET tests if only the empty stub is loaded: aviatesk/JET.jl#796 (comment) IMO JuliaRegistries/General#145354 was incorrect, the compat bounds of the latest releases are too strict.