Mysterious hang when using `tasty` with `hedgehog` and `IO`

[newhoggy]

When using tasty to run a hedgehog property that does simple IO (for example reading a file), the process can occasionally hang indefinitely.

The code for reproducing the hang can be found here: https://github.com/newhoggy/tasty-hang

The reproducer contains three executables:

• tasty-hang-bad - This executable uses tasty to invoke a hedgehog property that has withTests 1 and no generators, which effectively makes it a unit test. The test reads a small file. This can hang when run often enough.
• tasty-hang-good - This executable uses hunit instead of hedgehog to read a small file. This has not been observed to hang.
• tasty-hang-none - This executable invokes hedgehog directly form IO without using tasty at all. This has not been observed to hang.

I am running on Linux on WSL2 on Windows. The CPU has 32 cores.

To reproduce, I create a script file like this:

((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-1.log 2> ../stderr-1.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-2.log 2> ../stderr-2.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-3.log 2> ../stderr-3.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-4.log 2> ../stderr-4.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-5.log 2> ../stderr-5.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-6.log 2> ../stderr-6.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-7.log 2> ../stderr-7.log)
((date >&2; for x in {0..99999}; do echo "== $x =="; echo "== $x ==" >&2; tasty-hang-none; done) > ../stdout-8.log 2> ../stderr-8.log)

Then I run each loop concurrently like this:

cat script.sh | parallel

Then, in another terminal, I watch the progress of the run like this:

watch ls -la stdout-*.log

If any of the log files stop growing before the others, the associate process has hung.

This issue has been raised with hedgehog as well: hedgehogqa/haskell-hedgehog#547

[carbolymer]

@newhoggy You are accessing a non-existent file here https://github.com/newhoggy/tasty-hang/blob/main/tasty-hang-good/tasty-hang-good.hs#L22 - I don't know if that's making the tasty-hang-good reproducer falsely succeed. - fixed

Moreover, your reproduction script is writing outside of the reproducing repository directory.

I've used the repro.sh script and I've used the following reproduction steps to successfully reproduce hangs in tasty-hang-bad. On 32 cores, and n_loops=48 it takes few seconds to first blocked test suite to appear.

1. Use GHC 9.10.2, cabal 3.14.2, OS: Archlinux, kernel: 6.12.36-1-lts
2. Run ./repro.sh (contents below)
3. In another terminal window, in the reproducer repository run watch ls -la logs/stdout-*.log
4. Look for one of the log files to stop growing

repro.sh:

#!/usr/bin/env bash

run_hang() {
  cabal run tasty-hang-bad
}

# each invocation is doing a lot of IO, so you can set it to 1.5 * number of CPUs
n_loops=24

trap "echo 'Killing all loops...'; kill 0" SIGINT SIGTERM

mkdir -p logs
for i in $(seq 1 $n_loops); do
  (
    date >&2
    for x in {0..99999}; do
      echo "== $x =="
      echo "== $x ==" >&2
      run_hang
    done
  ) > ./logs/stdout-$i.log 2> ./logs/stderr-$i.log &
done

wait

echo "DONE!"

[Bodigrim]

tasty-hunit prefers to deal with errors on its own, but in the reproducer at least two Hedgehog routines catch all exceptions. Could you try to avoid them and see if the issue is still reproducible?

[newhoggy]

Which two routines are you referring to?

Do you mean I should replace evalIO with liftIO and also replace/remove something else?

Also this reproduces without tasty-hunit by using tasty-hedgehog instead.

[newhoggy]

This also reproduces on a Mac:

uname -a
Darwin MacBook-Pro-5.local 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:54:49 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6000 arm64

[Bodigrim]

I can reproduce using the script from #449 (comment) on macOS (aarch64) as well. All 24 threads will eventually hang, even the last one, so it is not a question of file system contention or extra high load. The hanging processes consume neither CPU nor memory, so presumably there is a deadlock somewhere.

Disabling exception handlers on Hedgehog side (as in void . H.check . H.withTests 1 . H.withShrinks 0 . H.Property H.defaultConfig $ do void . liftIO $ BS.readFile "README.md") does not change anything.

I'm unlikely to have an opportunity to investigate further any time soon. But please keep me posted.

[carbolymer]

I have replaced H.check invocation with the version which skips rendering of the results in tasty-hang-bad:

check2 :: MonadIO m => H.Property -> m Bool
check2 prop = liftIO $ 
  (== H.OK) . H.reportStatus
    <$> H.checkReport (H.propertyConfig prop) 1 (H.from 1) (H.propertyTest prop) (\_ -> pure ())

Which makes the tests not hang anymore.

Full code: https://github.com/newhoggy/tasty-hang/pull/2/files

Because tasty-hang-none is not causing any deadlocks, I suspect that there's some interaction between tasty's and hedgehog's rendering codes which is causing problems.

[Bodigrim]

@carbolymer that's interesting! Well, tasty itself does not do anything fancy with stdout, especially when redirected to a file as in examples above. But hedgehog employs concurrent-output (CC @joeyh) and it seems (at least after cursory reading) that it does quite a number of TMVar operations even before checking for hSupportsANSI.

[newhoggy]

Empirical evidence that check from hedgehog has a hanging problem in this comment: hedgehogqa/haskell-hedgehog#547 (comment)

[newhoggy]

The mystery deepens: hedgehogqa/haskell-hedgehog#547 (comment)

[Bodigrim]

@newhoggy given that you are fixing it in hedgehogqa/haskell-hedgehog#550, is there anything to do on Tasty side? Shall we close the issue?

[Bodigrim]

Eventually this appears to be a bug in GHC (see https://gitlab.haskell.org/ghc/ghc/-/issues/26324 and https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14734), nothing to fix in tasty.