Should use doctest's --no-magic to stop it from interfering

[DanielG]

doctest seems to do all manner of weird things with the GHC flags passed to it, see https://github.com/sol/doctest/blob/6daf5b9623001ad8ff30e59a7a8add2840f1cac7/src/Run.hs#L57. This can be disabled by passing --no-magic to doctest. With that flag everything seems to work even with new-build :)

FYI that flag was added in 0.11.3 I think, see sol/doctest@132e1b9.

[phadej]

You can use

x-doctest-options: --no-magic

I'm not sure I want to have it as default, have you an example package where it causes problems?

[DanielG]

Yeah I thought having it as a default might be a bit much which is why I didn't just shoot a PR. I think we should warn users about this in the readme/module docs regardless.

Here's a minimal reproducer: https://github.com/DanielG/cabal-doctest-repro. Looks like doctest tries to expand "directories" in the GHC flags to be all the files within them, but here the GhcMod argument it gets is actually both a directory and a module name so it adds the files within the directory to the GHC flags as well even though they are already there and GHC gets horribly confused.

$ cabal new-test doctest
Resolving dependencies...                 
In order, the following will be built (use -v for more details):
 - ghc-mod-5.9.0.0 (first run)
[1 of 1] Compiling Main             ( /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/setup/setup.hs, /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/setup/Main.o )
Linking /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/setup/setup ...
Configuring ghc-mod-5.9.0.0...
Preprocessing test suite 'doctest' for ghc-mod-5.9.0.0..
Building test suite 'doctest' for ghc-mod-5.9.0.0..
[1 of 2] Compiling Build_doctests   ( /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/doctest/autogen/Build_doctests.hs, /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/doctest/doctest-tmp/Build_doctests.o )
[2 of 2] Compiling Main             ( test/doctests.hs, /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/doctest/doctest-tmp/Main.o )
Linking /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/doctest/doctest ...
Preprocessing library for ghc-mod-5.9.0.0..
Building library for ghc-mod-5.9.0.0..
[1 of 2] Compiling GhcMod.Exe.Boot  ( GhcMod/Exe/Boot.hs, /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/GhcMod/Exe/Boot.o )
[2 of 2] Compiling GhcMod           ( GhcMod.hs, /tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/GhcMod.o )
Running 1 test suites...
Test suite doctest: RUNNING...

<no location info>: error:
    module ‘main:GhcMod.Exe.Boot’ is defined in multiple files: /tmp/cabal-doctest-repro/GhcMod/Exe/Boot.hs
                                                                GhcMod/Exe/Boot.hs
-i
-i/tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/autogen
-i/tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build
-i/tmp/cabal-doctest-repro
-hide-all-packages
-no-user-package-db
-package-db=/home/dxld/.cabal/store/ghc-8.2.2/package.db
-package-db=/tmp/cabal-doctest-repro/dist-newstyle/packagedb/ghc-8.2.2
-package-db=/tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/package.conf.inplace
-optP-include
-optP/tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/build/autogen/cabal_macros.h
-package-id=base-4.10.1.0
-package-id=doctest-0.11.4-64bebfa68ae243a0316af432873a28547a1b974ac35b8ec4d481ffb02260e85a
GhcMod.Exe.Boot
GhcMod
Test suite doctest: FAIL
Test suite logged to:
/tmp/cabal-doctest-repro/dist-newstyle/build/x86_64-linux/ghc-8.2.2/ghc-mod-5.9.0.0/test/ghc-mod-5.9.0.0-doctest.log
0 of 1 test suites (0 of 1 test cases) passed.
cabal: Tests failed for ghc-mod-5.9.0.0.

What use cases do you envision for doctest doing its magic when we ought to know better, being more closely integrated with cabal and all? The "magic" seems to just involve making it possible for the user not to have to know too much about what cabal is doing.

[phadej]

I see.

• I'd be +1 for having --no-magic as default, but there's no reverse flag in doctest (if for some very weird reason someone needs the magic?)
• otherwise I'm happy to accept a PR adding x-doctest-options: --no-magic to README and example packages.