Contributor-Guide: Tips for Using CI

[vinniefalco]

We need a section on how to use GHA, Drone, et. al. for Continuous Integration and especially on how to minimize the amount of resources used so that more targets can be tested. Boost authors and CPPA staff need to help with this... Peter Turcan cannot be expected to write this all on his own.

[alandefreitas]

Yes. Definitely.

Here's how I see the problem, the solution, and the guide we should write.

The problem

We first need to delimit the problem we're solving.

Ramey's problem

About Ramey's complaints, I had a look at his repositories. His serialization "Run tests" step takes more than 2 hours. You can serialize the whole BTC blockchain in two hours. This is basically a bug from my point of view. There's nothing we can do in CI to help him. He needs to define what's important and, depending on the case, define the quick tests exploring the API and the long tests exploring all fixtures the library can handle.

The real problem

For most other libraries, GHA and drone times are OK. The only problem is the time boostorg waits for runners to become available. In a way, this can be fixed by having a fork where the tests are run before pushing to boostorg. But this is still the main problem I believe we're trying to fix because we can't rely on tests that run on forks all of the time.

The solution

So then we have to delimit the solutions we have.

Drone

One solution is drone, which I believe people don't opt-in because of the cost of learning and maintaining when they already have their github workflow. Learning it is not that easy and some things happening under the hood are still a mystery even to me. Even Peter had some resistance to using drone and he now uses custom scripts that bypass the part of the infrastructure the CppAlliance provides (For instance, see https://github.com/boostorg/variant2/blob/develop/.drone.jsonnet). Then there's the maintenance cost since we can't just use actions in drone. For instance, the superproject uses the softprops/action-gh-release and site-docs uses jakejarvis/s3-sync-action. I have no idea what a steps of that kind would look like in drone. It would take a lot of effort to learn the tools (including drone itself, AWS, GitHub CLI) and cooperation with Sam to get that working. In the end, we would still have to rewrite and maintain everything at least twice. Once for the posix .sh and once for the windows .bat.

GHA runners

The second potential solution is cppallliance providing GHA self-hosted runners. These runners should have a default timeout of one hour or something so that people can only join them if they manage to get their tests working properly. Exceptions would only happen for benchmarks like unordered. About activating these runners when GitHub's shared runners are
over some threshold of load, I believe this is not so important at this point because the shared runners are always over whatever threshold as far as boostorg is concerned. So any extra cost we have with GHA is, in principle, the same cost we would have if people were using drone.

The guide

Considering these solutions, we should write a guide to help people set up CI. We must document drone and GHA, even if we don't provide GHA runners because this is what people are actually using and there should be good guidelines. I think it should have the following structure:

General guidelines

I think the most important point here, which should be the introduction, is what workflows people need to test, regardless of the CI system being used. That is: the cmake workflow from the boost root dir should work, add_subdirectory(${SELF}) should work, find_package(Boost COMPONENTS ${SELF}) should work, and b2 libs/$SELF should work.

Drone guidelines

After what should work regardless of the CI system, regarding drone, there's no much we can reuse from boostorg/boost-ci. boost-ci is a great template but documentation of the contents in CI is very poor. There's a lot of legacy stuff in the ci directory and the best we have is some comments in scripts. My feeling is those scripts are kind of a black box with no clear input and output parameters. They also make lots of assumptions about how things are organized like the layout of directories and the steps are not clearly delimited like in our new Boost.URL workflow where it's clear what each step is testing. So we just need to describe what the common setup is for the initial configuration: drone.star describes the configurations, which should probably use functions, and the rest should go in scripts drone.bat and drone.sh that will run the "everything" step. I don't think the generator should be part of this guide for now.

GHA guidelines

Regarding the instructions for GHA, we should include instructions that use the actions I provided for boost.url. The pattern in Boost.URL makes it very clear what workflows we're testing and moves any complexity to the actions so that it works all of the time and users don't have to worry about adjusting details related to the platforms. Then we can be sure all libraries just work when they are pulled as part of the super project and they're using caching and everything. Now that these actions have been battle tested, I have been moving them to a separate repository where they are properly documented. GHA doesn't care that much about where the actions are and we already use actions from various sources completely unrelated to boostorg. We could move these actions to cppalliance but I'm not sure it's worth moving them to boostorg, because that would create a lot of unnecessary discussions and they're not strictly related to boost anyway.

Containers

One more thing we could attempt to provide is containers. I don't know exactly how the process of sharing works, but the longest tests we have are basically because there are some containers that have to install the world before getting started. Some things are just things we obviously need, like the compiler. I don't see a good reason to pull a container without a compiler just to install the compiler on it. I think @anarthal did a good job on his workflows where the containers come with everything he needs.

[anarthal]

I like bullet pointing the world, so here we go:

• I completely agree that we need a guide.
• I've been looking at Ramey's stuff and he's doing zounds of run tests, plus there's a lot of warnings in the output - in my opinion, having a single executable with your test suite (using a test framework) is usually better. But it may not provide enough speedup, I haven't measured.
• I strongly believe that containers are the only sane way of getting quality CI builds. In my opinion, any CI that spells apt install is wasting resources. Containers give you build reproducibility, and depending on how you do it, local reproducibility. They work great for integration tests and work for different architectures. The only limitation is that there are no OSX images, so these builds still need the "manual" approach.
• Container sharing is ultra easy. You all could use my containers - they're public under the automation account https://github.com/anarthal-containers?tab=packages. If a container doesn't suit you, you can always "inherit" from it and create a new container that suits your needs (e.g. installing an additional package).
• If container builds is something the C++ alliance is interested on, I'm happy to work on it. My idea would be to generalize the ones I've got for MySQL (removing a couple of MySQL-specific things) and to create additional containers to cover the wide range of compilers other libraries test on (I did a selection; URL covers some more compilers than MySQL).
• As you may know, I don't use Boost.CI. I wrote my thing. I got the same feeling of poorly documented, legacy code than @alandefreitas. Boost.CI had also limitations, like not allowing me to run services I need for integration tests (strongly required in my case). It didn't support the CMake flows out of the box, at least when I last looked at it. I wrote a Python script, because then I didn't have to duplicate all the code for Windows and Unix. If I had to write something like Boost.CI, I would create a Python package with utility functions (e.g. install_boost, run_b2, run_cmake_subdir_tests), which we could get in PyPI and then install in our builds (or containers). Once again, if you feel this is something that could interest anyone, reach me out, I'm happy to work on it.
• Regarding the guide - if it's gonna cover how to use Boost.CI, I'm not the person. If it's covering any other stuff I know, I'm happy to help (or even write it myself). I think we need to make a decision on what we're going to recommend to our people before moving forward.

[vinniefalco]

Boost.CI is a stepping stone; it is a mile marker on the highway to CI nirvana. The modern version of Boost.CI would be a collection of GHA actions specific to Boost to make it easy to express CI activity for Boost libraries.

Regarding the guide - if it's gonna cover how to use Boost.CI, I'm not the person. If it's covering any other stuff I know

The guide should cover anything that is helpful. It could start by explaining what CI is, what services are available (some free, some paid), we probably want to explain the Boost Test Matrix. Eventually we want to explain Boost.CI in a sub-chapter, since so many repositories use it.

[anarthal]

Boost.CI is a stepping stone; it is a mile marker on the highway to CI nirvana. The modern version of Boost.CI would be a collection of GHA actions specific to Boost to make it easy to express CI activity for Boost libraries.

Does this mean we're moving away from Drone?

we probably want to explain the Boost Test Matrix

I definitely need this guide

[alandefreitas]

any CI that spells apt install is wasting resources.

Yes. That just reminded me of another problem we frequently have. The apt-get step fails very often just because of intermittent networking issues. In github we can ask it to rurun failed jobs and then it works, but that takes a lot of time because we have to wait for all other jobs to complete and then wait for available machines again. The jobs with lots of apt-get installs are often the ones we need to validate the most.

It may seem like that's not relevant, but if the probability of an intermittent issue is only 1% and there are 20 jobs, the probability of an intermittent apt-get issue in one of the jobs is 18%.

You all could use my containers

That would be good. We talked about that once but I couldn't figure out how to do it by myself. We could use your container and later we could discuss that a little better because we'll eventually need some more customized containers and I don't want to be bothering you with this all the time. In any case, I think we need one container per-compiler (or one container with all compilers?). A few apt-get installs after that should be OK. It's taking 4 minutes everything to redownload the compiler that's bothering me the most. Later we can adjust that to include LLVM+Clang for mrdox.

If I had to write something like Boost.CI, I would create a Python package with utility functions

Yes. Maybe it would be more composable if we just forwarded everything to python. The "everything" step that results from that and the lack of actions is still going to be a problem. I think GitHub actions are so nicely integrated into the interface, are easy to compose, run faster than drone once the machines are available, and have a zero marginal learning cost. That puts drone in a difficult position that no one would want to use it unless the benefits were immense. The equivalent of actions in a drone workflow is a plugin, but they are very limited and rely on running a secondary container, which is often not worth the cost.

For instance, it took us a few minutes to integrate caching in Github actions. It took us weeks in drone to rewrite lots of independent bat/sh scripts and we still find bugs in our scripts. The final result is still slower.

we probably want to explain the Boost Test Matrix

Oh! That's important.

[vinniefalco]

Does this mean we're moving away from Drone?

It means that there is nothing set in stone and we are always evaluating our current solutions and potential new solutions in order to improve things.

[Lastique]

Yes. That just reminded me of another problem we frequently have. The apt-get step fails very often just because of intermittent networking issues.

This depends on where the packages are being installed from. All stock Ubuntu repositories are cached by MS (I'm assuming, by GHA infrastructure or Asure), so installing packages from there is fast and problem-free. Pulling e.g. from official clang repositories is problematic because, besides the longer and more fragile network path, clang repos also have a download limit that may trigger if there are many CI jobs running. This means that you should generally avoid installing packages from external repositories and look for compiler versions that are available in stock repositories of various versions of Ubuntu. This has worked well for me, and I almost never see CI jobs fail due to network issues.

I believe, the same consideration is true for Docker container images. If the image is cached then running this container is fast and problem-free. If not, at the very least performance will suffer. I'm not sure what images are cached, if any, and whether that set of cached images is fixed or dynamic. However, I would like to note that a default Ubuntu image is not enough to run Boost tests and requires more configuration (including installing more packages) than running tests directly in the runner.

[alandefreitas]

@Lastique That's an incredibly useful comment. Indeed, one GHA workflow where this happens a lot is one that uses a container for Clang 3.8, which is installed with apt-get. We do have to come up with a common list of containers that come with the compilers, like the ones @anarthal developed.

[vinniefalco]

@sdarwin how does this work with the "cached repositories" when we use our own GHA runners?

[anarthal]

Does this mean we're moving away from Drone?

It means that there is nothing set in stone and we are always evaluating our current solutions and potential new solutions in order to improve things.

With @alandefreitas comments, the direction looks clearer to me now.

Please bear in mind that the default GitHub-hosted runners can't run Windows containers - this being one of the reasons I went for Drone. If we're getting self-hosted runners, I'm in for the new vision.

[anarthal]

I believe, the same consideration is true for Docker container images. [...]

I'm hosting my containers in the GitHub container registry and it works pretty well. If we're moving stuff to our own infrastructure (say AWS), I would place the containers in whatever container registry AWS has. That's the neat thing about containers - they live where you decide :)

[alandefreitas]

GitHub-hosted runners can't run Windows containers

Oh! I never heard about that before. Are there any workarounds? Does @sdarwin know about that?

[sdarwin]

GitHub-hosted runners can't run Windows containers

Here is a counterpoint to that issue. If you're running somewhat customized containers on ephemeral windows runners, then at each runner launch it has to download the docker image which is GBs in size. Our 2022 windows image is 8.5GB.

Instead of containers, simply using the preinstalled copy of visual studio, and download any other requirements, might be 100 times less network bandwidth consumed, it depends on the situation and what is being installed.

[sdarwin]

caching ... when we use our own GHA runners?

• As with Drone, pre-built VMs can have docker images baked in. An official image such as ubuntu:22.04 will be updated at the source though, and end up getting re-downloaded, so it would be difficult to cache.

• As with Drone, ubuntu packages are installed from ubuntu. Alan's statistic about 18% of drone jobs are failing seems a bit high. It's a topic worth consideration. How more caching could be done.