Prefer an alcotest extension mechanism for CIs

[dinosaure]

At present, alcotest recognises CIs because the maintainers want it to and adds a non-exhaustive, fixed list of CIs to be recognised. This poses several problems:

1. The choice of CIs supported is very specific (Jenkins does not appear, for example...)
2. Adding a CI means making a new Alcotest release
3. The maintainers have to support possible changes to the various CIs in order to guarantee stable Alcotest behaviour.

It is certainly more ambitious but perhaps more interesting to decouple CI support from the alcotest core. We could imagine a bechamel-style extension mechanism in order to:

1. not be restrictive on the support of CIs by alcotest
2. unblock certain possible behaviours of certain CIs by taking their specific features into account in these extensions.
3. decouple the release cycle of the core and these extensions

[MisterDA]

The following are my personal views on the subject.

In the FOSS ecosystem of OCaml, I haven't seen other CIs than GitHub Actions, ocaml-ci, opam-repo-ci (checking submissions to opam-repository), opam-health-check (periodic checks of the opam-repository), Gitlab CI, App Veyor. With a quick search, I've found only GitHub Actions offering some integration by parsing the logs, namely Workflow commands for GitHub Actions. Nothing related for Gitlab CI, and Jenkins seems not to have a dedicated syntax for reports, but rather is able to parse the (standardized and not) output of various tools. It seems that AppVeyor users can upload tests in the jUnit format to a predefined endpoint.

IMO, the most simple CI integration would be to display the test errors and retain build logs (people might want to extract them, or ssh into the CI…). That's given by the --show-error option, not the --verbose which doesn't retain logs on disk.
I can't recall if colors detection is also based on the CI variable or on the INSIDE_DUNE env var. CIs redirect Alcotest output, so it could also unconditionally enable colors.

Some (most?) CI systems define the CI environment variable to a truthy value. We can use it to enable the --show-error behavior in that case. I didn't do it because I was worried it would break the behavior on systems I didn't have some sort of control on, but it seems to be the way forward.

To address your first points:

1. yes, but what do most Aclotest users use?
2. Alcotest has followed its own development pace, CI patches are just patches. Why would that be a problem?
3. I've tried to ask for a feature in GHA over a year ago. CIs never change.

I think extensions are an overkill. Besides, how would Alcotest as-is detect the correct extension to install? If the burden falls on the user, that's also a problem for zero-config CIs. (oh, my bad, I'll look at the extensions in bechamel).

IMO, if you're feeling uneasy about all this, we can keep detecting the CI=true env var and enable --show-errors in that case, keep a light GitHub Actions integration, and remove detection of CIs for which there's no particular integration to have. I believe this is a reasonable compromise that doesn't weight much on the maintainers' shoulders or Alcotest development.

…

Months later I now recall that I went this way because it was much, much simpler than adding a printer for a normalized test format, such as jUnit or (better) Test Anything Protocol (tap) that can be easily consumed by CIs or CI plugins. You could say I was targeting the poor man's CI integration.

[dinosaure]

Thank you for all this explanation (with a very good overview). What bothers me most is the specificity of all these CIs that we have to maintain in Alcotest. This means that someone (you?) has to look after and maintain the details over the long term. This maintainance is also in full knowledge of what is happening in ocaml-ci/opam-ci/etc. which is not the case for all maintainers. What also bothers me is the imposed list of CIs that Alcotest manages (and if tomorrow, I made a CI, would you want to integrate it into Alcotest?): this list reflects a certain usage that is not that of "everyone" (I mentioned Jenkins but we can talk about CircleCI too).

In short, these aren't very interesting questions and they add extra work: something that neither you nor I have time to do.

The idea of extensions doesn't seem so bad to me if you think carefully about its integration: it will inevitably require the user to add a dependency for the CI but that's not a big deal. As far as zero-configs CI is concerned, it's not so "zero-config" if you have to modify Alcotest to get a nice output ;) . Finally, the CI environment variable seems to me to be the best compromise.

EDIT: Don't consider your initial PR to be blocked by this issue, as there can be several tempolarities in Alcotest management.