Include files without testsets ?

[ederag]

Trying to use TestPicker for Pluto,

test> cell_disabling

some tests fail because some functions are missing, for instance cleanup:
https://github.com/fonsp/Pluto.jl/blob/b9cff27e7b9e060cfeac2a1e31dee83af73f0e87/test/helpers.jl#L162

Is there any way to include helpers.jl in the module that is created to hold the tests ?
I managed to select either helpers or cell_disabling, but not both.

Tried with [a64165b9] TestPicker v1.1.2.

EDIT: I overlooked this in the README:

however it expects that every test file you have under your test/ folder is self-contained!

So please feel free to close.

[theogf]

Hey @ederag , I just implemented a new interface for TestPicker that allows to define custom testing. In particular you could define a PlutoTestInterface with a definition

TestPicker.preamble(::PlutoTestInterace) = :(include("helpers.jl"))

[ederag]

Thanks, looks great !
It should even be possible to add a TestPickerExt extension in Pluto to make this automatic.

But it does not work here yet (looks like "helpers.jl" is not included inthe module) :

(@v1.10) pkg> update TestPicker
  [a64165b9] ↑ TestPicker v1.1.3 ⇒ v1.2.1

(@v1.10) pkg> activate -
  Activating project at `~/share/prog/julia/dev/Pluto`

julia> using TestPicker

# Define a custom interface
julia> struct PlutoTestInterface <: TestBlockInterface end

# ... implement specialized methods ...
julia> TestPicker.preamble(::PlutoTestInterface) = :(include("helpers.jl"))

# Register with TestPicker
julia> add_interface!(PlutoTestInterface())

test> cell_disabling.jl
...
  Got exception outside of a @test
  UndefVarError: `cleanup` not defined
# `cleanup` is defined in "helpers.jl"

julia> TestPicker.INTERFACES
2-element Vector{TestBlockInterface}:
 TestPicker.StdTestset()
 PlutoTestInterface()

[theogf]

Right I would expect that the test was fetched by StdTestset(). I guess I should add a replace_interface! that would remove any existing interface.
For now you could just do popfirst!(TestPicker.INTERFACES).

[ederag]

Done,

julia> TestPicker.INTERFACES
1-element Vector{TestBlockInterface}:
 PlutoTestInterface()

but that did not help.
Also tried in a fresh session (in case there were a cache), pop before adding the interface, idem.

[theogf]

I thought that maybe you needed the absolute path on helpers.jl but you would get an error if it was that. I will give it a try later!

[theogf]

Ah I know why this does not work! The interface is for the testsets only...

I can confirm that running a testset from the file you mention works with the interface you wrote.

I will try to think of a way to make it work for test files as well.

[ederag]

Here the "fuzzy search" advertised in the README does not happen,
and it does not seem possible to select a specific testset either,
only typing "cell_disabling" Enter displays the file content,
and a further Enter runs the test file.
For instance

julia> using TestPicker

test> Disabling

Does nothing (and typing Enter at the end just show "Selecting test file(s) \n 0/48 (0)").

Tested with both julia 1.10.9 and 1.11.5.
Would you like another issue for that ?

[theogf]

To select a specific testset you need to include : into your first query, i.e.

test> disabling:

[ederag]

julia> using TestPicker

julia> struct PlutoTestInterface <: TestBlockInterface end

julia> TestPicker.preamble(::PlutoTestInterface) = :(include("helpers.jl"))

julia> add_interface!(PlutoTestInterface())
2-element Vector{TestBlockInterface}:
 TestPicker.StdTestset()
 PlutoTestInterface()

julia> popfirst!(TestPicker.INTERFACES)
TestPicker.StdTestset()

# Note the space after ':', typed before Enter, otherwise one has to type Enter twice
test> disabl: 
┌ Error: Could not complete test picker action due to error:
│ 1-element ExceptionStack:
│ `istestblock` must be implemented for type PlutoTestInterface.


# adapted from https://theogf.dev/TestPicker.jl/stable/#TestPicker.TestBlockInterface
julia> function istestblock(::PlutoTestInterface, node::TestPicker.SyntaxNode)
           return istestblock(TestPicker.StdTestset(), node)
       end
istestblock (generic function with 1 method)

test> disabl: 
┌ Error: Could not complete test picker action due to error:
│ 1-element ExceptionStack:
│ `istestblock` must be implemented for type PlutoTestInterface.


# with all function names fully qualified
julia> function TestPicker.istestblock(::PlutoTestInterface, node::TestPicker.SyntaxNode)
           return TestPicker.istestblock(TestPicker.StdTestset(), node)
       end

test> disabl: 
┌ Error: Could not complete test picker action due to error:
│ 1-element ExceptionStack:
│ `blocklabel` must be implemented for type PlutoTestInterface.


julia> function TestPicker.blocklabel(::PlutoTestInterface, node::TestPicker.SyntaxNode)
           return "pluto: " * JuliaSyntax.sourcetext(JuliaSyntax.children(node)[2])
       end

test> disabl: 
┌ Error: Could not complete test picker action due to error:
│ 1-element ExceptionStack:
│ UndefVarError: `JuliaSyntax` not defined


julia> function TestPicker.blocklabel(::PlutoTestInterface, node::TestPicker.SyntaxNode)
           return "pluto: " * TestPicker.JuliaSyntax.sourcetext(TestPicker.JuliaSyntax.children(node)[2])
       end

test> disabl: 
...
>  [ Info: Executing testblock pluto: "Indirectly disabled cell + usings (#3263)" from cell_disabling.jl:427──────────────────────────────────────╯
Test Summary:                             | Pass  Broken  Total  Time
Indirectly disabled cell + usings (#3263) |    5       1      6  3.1s

Nice!
I'd just have expected istestblock and blocklabel to have fallbacks.

[theogf]

I guess I could make fallbacks, the primary use was really to be able to use different blocks like @testitem.
I think in this case it probably make more sense to generally just have a different preamble. The use of the TestBlockInterface is a bit hacky and does not work for test files.
I will think about a different way to approach the problem.