BazelCI failing for rules_haskell with release Bazel since Dec 19

[c-parsons]

Example buildkite failure:
https://buildkite.com/bazel/rules-haskell-haskell/builds/578#ec763d12-7c1e-4037-8ffb-b7937c361ce7

Unfortunately I don't have any insight into the error message.

Snip:

exe/Main.hs:671:24: error:
--
  | • Exception when trying to run compile-time code:
  | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/54c8b5e2d2a3cfdab9fa69c8a71fac71/execroot/rules_haskell/external/rules_haskell_ghc_linux_amd64/bin/../lib/settings: openFile: does not exist (No such file or directory)
  | Code: makeGhcVersionChecker (pure <$> getLibdir)
  | • In the Template Haskell splice
  | $$(makeGhcVersionChecker (pure <$> getLibdir))
  | In the expression: $$(makeGhcVersionChecker (pure <$> getLibdir))
  | In an equation for ‘ghcVersionChecker’:
  | ghcVersionChecker = $$(makeGhcVersionChecker (pure <$> getLibdir))
  | \|
  | 671 \| ghcVersionChecker = $$(makeGhcVersionChecker (pure <$> getLibdir))
  | \|                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[aherrmann]

Looking through the history it looks like the first instance of this failure occurs after merge of #1458. However, that PR seems unrelated to the issue. To be sure I started a pipeline with that commit reverted, but that fails as well.

The first failing pipeline is here and the last succeeding one is here. So far, all bazel-CI runs since have consistently failed with this error. The docker image and bazel version between the two seems to be identical. The docker command and environment seem to be slightly different though I don't know if that's relevant.

---

  | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/54c8b5e2d2a3cfdab9fa69c8a71fac71/execroot/rules_haskell/external/rules_haskell_ghc_linux_amd64/bin/../lib/settings: openFile: does not exist (No such file or directory)

Testing on a local build in bindist mode the file exists as expected

$ ls $(bazel info output_base)/external/rules_haskell_ghc_linux_amd64/bin/../lib/settings

[mboes]

Anecdotally, I've seen the error @aherrmann mentions before in CI builds of a feature branch. It baffled me at the time because I couldn't see what caused it, and then it disappeared anyways. Hard to find now, because BuildKite doesn't offer a build log search feature AFAIK.

[mboes]

Found a failing build: https://github.com/tweag/rules_haskell/runs/1677520752. This is from #1471.

[aherrmann]

Looking at bazel aquery 'mnemonic(HaskellCabalBinary, @ghcide//:_ghcide_exe_ghcide)' the settings file is indeed not an input to the action. In fact it doesn't seem to be tracked anywhere in rules_haskell. I'm not sure why this wasn't an issue before, but the error looks legitimate and we should track the settings file and add it as an input to probably all actions using GHC.

[mboes]

I'm as surprised as you that this wasn't an issue before, and it's unclear what changed to make this an issue now. But I'm also surprised that we get as far as not finding the settings file. In the bindist case, shouldn't the libraries that GHC binaries depend on also be inputs? The ghc binary dynamically loads a number of .so's in the GHC installation folder, i.e. the same place the settings file resides. Those don't appear to be part of the inputs either.