[WIP] Add performance testing

[evrardjp]

Without this, we are not comparing our BCI to UBI.
This is a problem, as we are not proving the value of our
efforts, from a technical standpoint.
We should show why our containers are probably better from
a technical perspective (or show where we can improve).

This fixes it by adding multiple builds of of the same
application, in 3 different runtimes: BCI, SLE current default
image, and UBI from RH. To make it significant, I chose
to use software which makes sense for ISVs or SUSE
Hub, so I chose grafana.

The grafana UBI files come from RH dockerfile in https://catalog.redhat.com/software/containers/rhel8/grafana/5edf9c35dd19c7063a62aff6?container-tabs=dockerfile

Closes: #41

[evrardjp]

self = DockerRuntime(marks=None, build_command='docker build .', runner_binary='docker', _runtime_functional=True), stdout = ''

    def get_image_id_from_stdout(self, stdout: str) -> str:
        # docker build prints this into the last non-empty line:
        # Successfully built 1e3c746e8069
        # -> grab the last line (see podman) & the last entry
>       last_line = list(
            filter(None, map(lambda l: l.strip(), stdout.split("\n")))
        )[-1]
E       IndexError: list index out of range

That's because stdout is empty:

CommandResult(command=b'docker build .', exit_status=0, stdout=None, stderr=b'#1 [internal] load build definition from Dockerfile\n#1 sha256:8b3833760bbc33cf0e9a2a352d88447990e297935136e77284eb85d183812c72\n#1 transferring dockerfile: 75B done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:6aa514c4bbf2349aa16efaca71cf742ec1b916d5b0c1076bfa60ca49597ab213\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for docker.io/library/ubuntu:bionic\n#3 sha256:42b24fe0142b829277f500329d7c50fc0d654412fc6d296283027ac72a38314d\n#3 DONE 0.7s\n\n#4 [1/2] FROM docker.io/library/ubuntu:bionic@sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8\n#4 sha256:fd6c9ed36b7ae183be04e452dbad1302f769da67a97754b8644470bc461a059c\n#4 DONE 0.0s\n\n#5 [2/2] WORKDIR /TODO\n#5 sha256:27f20393f7b82d6f80b1f8623c952ae9cb3a6c794aafcd6c7925101715235fe8\n#5 CACHED\n\n#6 exporting to image\n#6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#6 exporting layers done\n#6 writing image sha256:0062df290c70333608a837ae460300c1fc3d7cf60e248ae4cb5bdef7b4559a0b done\n#6 DONE 0.0s\n') and vars: {'exit_status': 0, '_stdout_bytes': b'', '_stderr_bytes': b'#1 [internal] load build definition from Dockerfile\n#1 sha256:8b3833760bbc33cf0e9a2a352d88447990e297935136e77284eb85d183812c72\n#1 transferring dockerfile: 75B done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:6aa514c4bbf2349aa16efaca71cf742ec1b916d5b0c1076bfa60ca49597ab213\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for docker.io/library/ubuntu:bionic\n#3 sha256:42b24fe0142b829277f500329d7c50fc0d654412fc6d296283027ac72a38314d\n#3 DONE 0.7s\n\n#4 [1/2] FROM docker.io/library/ubuntu:bionic@sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8\n#4 sha256:fd6c9ed36b7ae183be04e452dbad1302f769da67a97754b8644470bc461a059c\n#4 DONE 0.0s\n\n#5 [2/2] WORKDIR /TODO\n#5 sha256:27f20393f7b82d6f80b1f8623c952ae9cb3a6c794aafcd6c7925101715235fe8\n#5 CACHED\n\n#6 exporting to image\n#6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#6 exporting layers done\n#6 writing image sha256:0062df290c70333608a837ae460300c1fc3d7cf60e248ae4cb5bdef7b4559a0b done\n#6 DONE 0.0s\n', '_stdout': None, '_stderr': None, 'command': b'docker build .', '_backend': <testinfra.backend.local.LocalBackend object at 0x7f0ec66e5b80>}

Here is the output of the build:

Looking better:
[+] Building 1.6s (8/8) FINISHED                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 92B                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                                                  1.3s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                     0.0s
 => [1/3] FROM docker.io/library/ubuntu:bionic@sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8                                            0.0s
 => CACHED [2/3] WORKDIR /TODO                                                                                                                                    0.0s
 => [3/3] RUN echo "woot"                                                                                                                                         0.2s
 => exporting to image                                                                                                                                            0.0s
 => => exporting layers                                                                                                                                           0.0s
 => => writing image sha256:f67fe10a7de209534d274c565291423b968190f0e4f39d8eb3cfc8012bed2509                                                                      0.0s

How do the build here differ from the previous one we are doing?

[evrardjp]

The docker build --no-cache didn't change a thing

[evrardjp]

does that work properly ? See also L56. (Or maybe build and run don't have the same kind of output?)

[dcermak]

It appears to do the right thing on my machine: no containers are left on the system with both podman and docker after running the performance tests.

[evrardjp]

Rebased, squashed a few commits together.

[evrardjp]

To be closer to the reality between UBI and BCI, we need to also have a base container built, or need to build both base containers in this test.

[evrardjp]

This is still a draft... More work needs to be done to compare the different containers:

• Making those container build and run for the differnet distros
• Discuss whether or not the pytest junit files are enough. If not, parse podman build results for more details/profiling.
• Add not only build time testing, but also other runtime/performance testing

[dirkmueller]

Closing this for now. Please don't hesitate to reopen if there is time to rebase and finish it off!