TestItems not isolated from each other

[KristianHolme]

If i understand the @testitems correctly, each item is supposed to be run isolated from the other tests, with only the Test package and the tested package being loaded. However, it seems like the order of execution of @testitems does matter, and that loaded modules in one item is also loaded for items run later.

MWE:

If i create an empty project called MyTestPackage and add TestItemRunneras a test dependency, as well as the ColorTypes package, these tests fail:

@testitem "Load test dep" begin
    using ColorTypes
    @test :ColorTypes in Symbol.(Base.loaded_modules_array())
end
@testitem "Not loaded test dep" begin
    @test (:ColorTypes in Symbol.(Base.loaded_modules_array())) == false
end

because the ColorTypes package is loaded in the second test. If the order of the two tests is switched, then both pass.

This becomes problematic when testing extensions, as i want to test the behaviour of some function both with and without the extensions trigger package loaded. When running tests using ]test in the REPL, a workaround is to just reorder the testitems so that the testitems that load the trigger package is run after the items that should not have the trigger package loaded. But when using the VS Code extension this is more of a problem since tests are run in parallel. Even if i test the entire package in the test pane in VS Code and all tests passes, running all tests again causes some tests to fail because the trigger package then was loaded in the previous run, and are still present in the test process (?).

Am I doing something wrong here, or is this intended/necessary behavior?

Package Versions:

Status `~/Code/MyTestPackage/test/Project.toml`
  [3da002f7] ColorTypes v0.12.1
  [f8b46487] TestItemRunner v1.1.0
  [8dfed614] Test v1.11.0

Julia Version 1.11.7