Improve integration test execution time

[datosh]

This topic was suggested in the community meeting April 16.

The goal is improve the execution time of the dockerized integration tests which are implemented using go testcontainers, with the primary focus of improving execution time in GitHub Actions and a secondary focus to document and improve execution time when running locally.

The execution time of the GHA based integration tests is currently 5-6min.

[datosh]

Analysis

Here is a breakdown of the execution time for the integration tests as we stand today. The measurements were taken on my local system. They are close to the GHA exection times, but primarily serve to help us understand where we spend the most time running integration tests.

Fresh System - 4min

On a fresh system where we ran docker system prune --all --volumes --force the integration test take about 4 minutes to execute.

ok      github.com/openpubkey/opkssh/test/integration   172.847s
OS_TYPE=ubuntu go test -tags=integration ./test/integration -timeout=15m  -v  4,60s user 2,31s system 3% cpu 2:53,92 total

Re-run - 60 seconds

Running the same tests again (without making any modification to the source code) gives us an execution time of 60 seconds.

ok      github.com/openpubkey/opkssh/test/integration   57.689s
OS_TYPE=ubuntu go test -tags=integration ./test/integration -timeout=15m  -v  4,59s user 2,21s system 11% cpu 58,768 total

Time breakdown per test

Structure of Integration Tests

The integration tests use containers to simulate different operating systems (Debian, CentOS & Arch) and spin up an OIDC provider. These 4 images are built as part of the integration test execution. The results of these builds are preserved and Docker caching is used where possible during the same and subsequent runs. This is explains why re-running the integration tests is faster.

This works on my local system, but GitHub runners are ephemeral!. Every pipeline has to rebuild the images from the base image, every time!

When we look at the time required to build the CentOS image, we can see that 20s are spend building the opkssh binary and 30s are spent downloading dependenices from the package manager. This looks similar for all 3 OS images.

CentOS Full breakdown (timings are all the way to the right)

 => [builder 1/6] FROM docker.io/library/golang:1.23@sha256:e54daaadd35ebb90fc1404ecdc6eb7338ae13555f71a71856ad96976ae084e44                             14.0s
 => [builder 2/6] WORKDIR /app                                                                                                                            0.7s
 => [builder 3/6] COPY go.mod go.sum ./                                                                                                                   0.0s
 => [builder 4/6] RUN go mod download                                                                                                                     5.8s
 => [builder 5/6] COPY . ./                                                                                                                               0.2s
 => [builder 6/6] RUN go build -v -o opksshbuild                                                                                                         17.3s
 => [stage-1  1/19] FROM quay.io/centos/centos:stream9@sha256:131cca863dfe05582de3cf5907518288e1c68d75bc4df3a3355472dc62427cb7                            5.6s
 => [stage-1  2/19] RUN dnf update -y &&     dnf install -y sudo openssh-server openssh-clients telnet wget jq &&     dnf clean all                      24.3s
 => [stage-1  3/19] RUN useradd -rm -d /home/test -s /bin/bash -g root -G wheel -u 1000 test                                                              0.2s
 => [stage-1  4/19] RUN  echo "test:test" | chpasswd                                                                                                      0.2s
 => [stage-1  5/19] RUN echo "test ALL=(ALL:ALL) NOPASSWD: ALL" | tee /etc/sudoers.d/test                                                                 0.2s
 => [stage-1  6/19] RUN useradd -rm -d /home/test2 -s /bin/bash -u 1001 test2                                                                             0.3s
 => [stage-1  7/19] RUN  echo "test2:test" | chpasswd                                                                                                     0.3s
 => [stage-1  8/19] RUN mkdir /var/run/sshd                                                                                                               0.3s
 => [stage-1  9/19] WORKDIR /app                                                                                                                          0.0s
 => [stage-1 10/19] COPY --from=builder /app/opksshbuild ./opksshbuild                                                                                    0.0s
 => [stage-1 11/19] COPY --from=builder /app/scripts/install-linux.sh install-linux.sh                                                                    0.0s
 => [stage-1 12/19] RUN chmod +x install-linux.sh                                                                                                         0.2s
 => [stage-1 13/19] RUN bash ./install-linux.sh --install-from=opksshbuild --no-sshd-restart                                                              0.3s
 => [stage-1 14/19] RUN opkssh --version                                                                                                                  0.2s
 => [stage-1 15/19] RUN ls -l /usr/local/bin                                                                                                              0.2s
 => [stage-1 16/19] RUN printenv PATH                                                                                                                     0.2s
 => [stage-1 17/19] RUN echo "http://oidc.local:9998/ web oidc_refreshed" >> /etc/opk/providers                                                           0.2s
 => [stage-1 18/19] RUN if [ -n "$BOOTSTRAP_POLICY" ] ; then opkssh add "test" "test-user@zitadel.ch" "http://oidc.local:9998/"; else echo "Will not ini  0.2s
 => [stage-1 19/19] RUN ssh-keygen -A                                                                                                                     0.5s

OIDC Full breakdown (timings are all the way to the right)

 => [1/6] FROM docker.io/library/golang:1.20.12-bookworm@sha256:796bca00a5c2167dc32ddaab1371246d4713a860dfe157cac21648b42818d3d2                         15.2s
 => [2/6] WORKDIR /app                                                                                                                                    0.5s
 => [3/6] RUN git clone --branch test https://github.com/bastionzero/oidc.git                                                                             1.5s
 => [4/6] WORKDIR /app/oidc/                                                                                                                              0.0s
 => [5/6] RUN go mod download                                                                                                                             3.0s
 => [6/6] RUN go build -o /server -v ./example/server/dynamic                                                                                            15.0s

Improvement Ideas

Force Stop Provider Container

I noticed the provider conatiner is slow to shut down, it always takes 10 seconds. This can be tested using

cd test/integration/provider
docker build -t oidc -f exampleop.Dockerfile .
CONTAINER_ID=$(docker run -d oidc)
time docker stop ${CONTAINER_ID}

Implemented by #150

Common Base Image

All SSH container images use the same go base image golang:1.23, but OIDC provider uses a different one golang:1.20.12-bookworm. Switch OIDC to use golang:1.23 so the same (already downloaded) image can be reused.

Implemented by #150

Cache Image Layers in GHA

As I mentioned earlier, each workflow in GHA gets a new environment, therefore it is not possible to make use of previously build image layers.

I have looked into setting up caching, but was not able to get a working solution.

It is possible to cache image layers in GHA using the official docker actions with buildx support, with the cache-to and cache-from arguments

But because we are using test containers and they do not support buildkit/buildx we cannot take advantage of that feature.

So we have to change the approach to prevent full rebuild on each run! One solution would be to build the images (outside testcontainers) and then run the produced image in the test.

Other Aeas

TestEndToEndSSHWithRefresh

This test is particularly slow because it waits 10 seconds for token to become invalid.
I don't understand the code well enough to make any suggestions on how to improve it, but maybe someone has an idea..?

[EthanHeilman]

Thanks for this analysis. Really helpful

But because we are using test containers and they testcontainers/testcontainers-go#573 we cannot take advantage of that feature.

Could we move away from test containers?

[EthanHeilman]

I'm running into occasional integration test failures where an integration test takes too long to run and is canceled.