tests: add fips.enable.tls

[HuijingHei]

To verify that using TLS works in FIPS mode by having Ignition
fetch a remote resource over HTTPS with FIPS compatible algorithms.

tests/containers: build the container based on nginx-126:10.1.
See Timothee's comment #4477 (comment)

Fixes https://issues.redhat.com/browse/COS-3487

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[gemini-code-assist]

Code Review

This pull request adds a test container fips-nginx to verify file downloads over HTTPS with FIPS-approved cryptography. A thorough security audit found no significant vulnerabilities, though a low-risk issue regarding input validation in build.sh was noted. However, a critical issue in the Containerfile will prevent the container from building, and several suggestions have been made to improve the build.sh script's robustness and cleanliness.

[HuijingHei]

Hold on, looks this can only test x86_64, as I just setup the env on x86_64. Do we want to run on all multiarch?

[travier]

Hold on, looks this can only test x86_64, as I just setup the env on x86_64. Do we want to run on all multiarch?

Hum, I don't see anything x86_64 specific here. This should work on all arches but I'm also OK testing just on x86_64 as the arch should not matter here.

[travier]

Thanks, this looks really good. The main nit that I have is that we should be running a RHEL based container as only those can be FIPS compliant.

Now we don't really want to leave credentials on the server on GCP so we would have to work around that by:

• Pulling the base image from https://catalog.redhat.com/en/software/containers/rhel10/nginx-126/677d3718e58b5a1ae5598058 on your system with Red Hat credentials
• Using it for a local build for the nginx container image with the certificates
• Exporting the container image as an OCI archive
• Uploading the OCI archive to the server on GCP
• Importing the OCI archive into the podman container storage

That should get us very close to a "compliant" deployment.

Edit: fixed the order to build your system and then upload the final image to the server on GCP.

[openshift-cherrypick-robot]

@HuijingHei: new pull request created: #4490

Details

In response to this:

/cherrypick rhcos-4.18
/cherrypick rhcos-4.17
/cherrypick rhcos-4.16
/cherrypick rhcos-4.15
/cherrypick rhcos-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@HuijingHei: new pull request created: #4491

Details

In response to this:

/cherrypick rhcos-4.18
/cherrypick rhcos-4.17
/cherrypick rhcos-4.16
/cherrypick rhcos-4.15
/cherrypick rhcos-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dustymabe]

Hey @HuijingHei.. I'm sorry I didn't pay attention to this PR when it was opened.

I'm interested to know if this test covers more than the test that was added in #4373 - i.e. do they test the same things? Is this new test really needed?

[HuijingHei]

I'm interested to know if this test covers more than the test that was added in #4373 - i.e. do they test the same things? Is this new test really needed?

A little different with #4373, this is to ensure Ignition can fetch a remote resource over HTTPS with FIPS compatible algorithms. Maybe @travier has more context about this.

Copy the steps from comment, hope this can be helpful.

• Setup an RHCOS or RHEL system in FIPS mode somewhere (maybe a small VM on AWS)
• Generate a certificate on the system to make sure it's generated in FIPS mode and uses only FIPS compatible algorithms
• Setup an HTTP server (nginx?) which serves a small file over HTTPS
• Make sure nginx only serves files over HTTPS with FIPS compatible algorithms
• Make a kola test that sets up an RHCOS system in FIPS mode and fetches the file from the HTTP server successfully.

[dustymabe]

this is to ensure Ignition can fetch a remote resource over HTTPS with FIPS compatible algorithms.

this implies that there are requirements on the remote server for properly testing FIPs. Is that correct? I don't see how just enabling FIPs on the CoreOS instance we are testing doesn't ensure FIPs compliant crypto algorithms were used.

[jlebon]

Wait, I'm confused why this needs a permanent test fixture VM running in the cloud. Couldn't the test just provision two VMs? See e.g. the tang test which is very similar (sets up a service on the first VM, then Ignition queries that service on the other)?

As for testing itself, I think we really want to test the negative case too. One idea to not actually pay for the cost of a separate test is:

• on the server side, we have two nginx instances: one which only accepts FIPS-compliant algorithms, and one which only accepts non-FIPS-compliant algorithms
• on the test side, we feed an Ignition which .config.merges another Ignition config from the FIPS-compliant nginx, and in that other Ignition config, we have a file resource request to the non-FIPS-compliant nginx.

Then we verify that the error is what we expect (failure to fetch the file resource).

[dustymabe]

Wait, I'm confused why this needs a permanent test fixture VM running in the cloud.

Yeah. That definitely is a question I have too.

Couldn't the test just provision two VMs? See e.g. the tang test which is very similar (sets up a service on the first VM, then Ignition queries that service on the other)?

probably.

As for testing itself, I think we really want to test the negative case too.

I think what you are saying is that #4373 covers the positive test case such that the test coverage with this fips.enable.tls test as merged in this PR is redundant with #4373.

But you are also saying that the work in this PR could be repurposed to set up a "negative" test with a web server that only allows non-FIPS-compliant algorithms and verify it fails?

Is all of that correct?

[HuijingHei]

why this needs a permanent test fixture VM running in the cloud. Couldn't the test just provision two VMs? See e.g. the tang test which is very similar (sets up a service on the first VM, then Ignition queries that service on the other)?

IMU, the permanent VM could make the test simpler, but I can try with two VMs.
Edit:
If using konflux to build the container, need to make the IP argument as const 10.0.2.2, does that make sense? I update the Containerfile that needs IP as argument like main...HuijingHei:coreos-assembler:hhei/improve-fips-tls
Or maybe can build the container on the first VM.

Another thing is registry.redhat.io/rhel10/nginx-126:10.1 does not include openssl, but we need it to generat a self-signed TLS certificate, so I use rhel9/nginx-126 instead.

[travier]

this implies that there are requirements on the remote server for properly testing FIPs. Is that correct?

Yes, FIPS is not just a configuration on the client. Both client & server have to be FIPS enabled. In #4373, I don't know what the server in AWS is used as a configuration and if it only accepts FIPS approved algorithms.

Wait, I'm confused why this needs a permanent test fixture VM running in the cloud. Couldn't the test just provision two VMs? See e.g. the tang test which is very similar (sets up a service on the first VM, then Ignition queries that service on the other)?

Yes, this might indeed be possible but it would be much more difficult as we need to run FIPS enabled RHCOS on both sides and pass credentials to pull a RHEL nginx container image or build it in CI where here we built it once.

As for testing itself, I think we really want to test the negative case too. One idea to not actually pay for the cost of a separate test is:

• on the server side, we have two nginx instances: one which only accepts FIPS-compliant algorithms, and one which only accepts non-FIPS-compliant algorithms
• on the test side, we feed an Ignition which .config.merges another Ignition config from the FIPS-compliant nginx, and in that other Ignition config, we have a file resource request to the non-FIPS-compliant nginx.

Then we verify that the error is what we expect (failure to fetch the file resource).

Having an negative test would indeed be good idea but I'm not sure about mixing both into a single one. That might make it harder to figure why it the test fails if there is an issue. Once we have the remote server, spinning one VM for each test isn't that much work.

[travier]

Ah, and it's not enough to generate the cert in CI, it has to be generated on a FIPS enabled system so it would have to be generated in the first VM.

So the flow would look like:

• Spawn a first FIPS RHCOS VM
• Have it generate FIPS compatible TLS certs with the IP that will be used
• Have it run registry.redhat.io/rhel10/nginx-126:10.1 (you don't need to bake the certificate in the image, you can pass them as volume mounts)
• Then spawn the second VM, pointing it to the first

[travier]

Another thing is registry.redhat.io/rhel10/nginx-126:10.1 does not include openssl, but we need it to generat a self-signed TLS certificate, so I use rhel9/nginx-126 instead.

We don't need to generate the cert in the container image, we can generate it anywhere as long as the host is FIPS enabled and then pass it to the container at runtime using volume mounts.

[HuijingHei]

Another thing is registry.redhat.io/rhel10/nginx-126:10.1 does not include openssl, but we need it to generat a self-signed TLS certificate, so I use rhel9/nginx-126 instead.

We don't need to generate the cert in the container image, we can generate it anywhere as long as the host is FIPS enabled and then pass it to the container at runtime using volume mounts.

Then in our CI env, how to save the generated cert files (e.g. fips-server.crt, fips-server.key)?

[dustymabe]

Yes, FIPS is not just a configuration on the client. Both client & server have to be FIPS enabled. In #4373, I don't know what the server in AWS is used as a configuration and if it only accepts FIPS approved algorithms.

But the client is configured to only request FIPs approved algorithms (otherwise the kernel would throw an error), right?

Let's say that you are right, both client and server need to be FIPs enabled and won't work if not. What do FIPs enabled RHEL servers do for updates? Do they reach out to Red Hat's services and does that work? If so then we can assume stuff from Red Hat is FIPs enabled and find a URL of some artifact from there to download? Then we don't need to set up the second VM (unless we really want to do the negative test).

FTR when I did #4373 I confirmed that the test failed prior to the Ignition fix so it's definitely testing something.

[jlebon]

Yes, this might indeed be possible but it would be much more difficult as we need to run FIPS enabled RHCOS on both sides and pass credentials to pull a RHEL nginx container image or build it in CI where here we built it once.

The nginx instance can serve the cert itself and the client Ignition config can fetch it via its .config.security.tls.certificateAuthorities section.

[travier]

Another thing is registry.redhat.io/rhel10/nginx-126:10.1 does not include openssl, but we need it to generat a self-signed TLS certificate, so I use rhel9/nginx-126 instead.

We don't need to generate the cert in the container image, we can generate it anywhere as long as the host is FIPS enabled and then pass it to the container at runtime using volume mounts.

Then in our CI env, how to save the generated cert files (e.g. fips-server.crt, fips-server.key)?

Either you generate them on the host where you're going to run the container (so this way it should be FIPS enabled), or you generate them once on a FIPS enabled RHCOS VM and directly include them here in the test. It does not matter if the key is not secret, that's not what we are testing.

[travier]

Yes, FIPS is not just a configuration on the client. Both client & server have to be FIPS enabled. In #4373, I don't know what the server in AWS is used as a configuration and if it only accepts FIPS approved algorithms.

But the client is configured to only request FIPS approved algorithms (otherwise the kernel would throw an error), right?

It's not the kernel that would fail but something in openssl. But yes, ideally that would be enough but I don't know enough to be sure.

I could only find https://www.ibm.com/docs/en/rpa/30.0.x?topic=2-enabling-fips:

For a fully compliant system, you must enable FIPS encryption in both the server and the clients' operating systems.

We'll have to ask Clement again.

Let's say that you are right, both client and server need to be FIPs enabled and won't work if not. What do FIPs enabled RHEL servers do for updates? Do they reach out to Red Hat's services and does that work? If so then we can assume stuff from Red Hat is FIPS enabled and find a URL of some artifact from there to download? Then we don't need to set up the second VM (unless we really want to do the negative test).

I don't know and that's a good idea indeed. There are likely FIPS enabled and publicly accessible servers at Red Hat somewhere.

FTR when I did #4373 I confirmed that the test failed prior to the Ignition fix so it's definitely testing something.

OK, so it would be good to try if the test fails with a FIPS nginx server but a non-FIPS RHCOS as Jonathan suggested.

[HuijingHei]

on the server side, we have two nginx instances: one which only accepts FIPS-compliant algorithms, and one which only accepts non-FIPS-compliant algorithms
on the test side, we feed an Ignition which .config.merges another Ignition config from the FIPS-compliant nginx, and in that other Ignition config, we have a file resource request to the non-FIPS-compliant nginx.

Post the test results:

Test side: start test VM with fips enabled

• Ignition config from the FIPS-compliant nginx successfully
• Ignition config from the non-FIPS-compliant nginx failed with error remote error: tls: handshake failure

Doing tests on server side with none FIPs enabled, or with FIPs enabled, the result is the same. So this proves Dusty is correct, it does not matter if server side is FIPs enabled or not. Or maybe I am wrongly understanding this.

use storage.files instead,

     {
        "path": "/var/resource/https-non-fips",
        "contents": {
          "source": "https://192.168.122.152:8443/index.html"
        }
      }

[   12.377872] systemd[1]: Starting Ignition (files)...
[   12.411306] ignition[1125]: Ignition 2.25.1
[   12.411932] ignition[1125]: Stage: files
[   12.412812] ignition[1125]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"
[   12.417427] ignition[1125]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"
[   12.439107] ignition[1125]: Adding "NON-FIPS Test Server" to list of CAs
[   12.502305] ignition[1125]: files: ensureUsers: op(1): [started]  creating or modifying user "core"
[   12.719450] ignition[1125]: files: ensureUsers: op(1): [finished] creating or modifying user "core"
[   12.720490] ignition[1125]: files: ensureUsers: op(2): [started]  adding ssh keys to user "core"
[   12.724459] ignition[1125]: wrote ssh authorized keys file for user: core
[   12.724876] ignition[1125]: files: ensureUsers: op(2): [finished] adding ssh keys to user "core"
[   12.726877] ignition[1125]: files: createFilesystemsFiles: createFiles: op(3): [started]  writing file "/sysroot/etc/ignition-machine-config-encapsulated.json"
[   12.729820] ignition[1125]: files: createFilesystemsFiles: createFiles: op(3): [finished] writing file "/sysroot/etc/ignition-machine-config-encapsulated.json"
[   12.730412] ignition[1125]: files: createFilesystemsFiles: createFiles: op(4): [started]  writing file "/sysroot/var/resource/https-non-fips"
[   12.732472] ignition[1125]: files: createFilesystemsFiles: createFiles: op(4): GET https://192.168.122.152:8443/index.html: attempt #1
[   12.768918] ignition[1125]: files: createFilesystemsFiles: createFiles: op(4): GET error: Get "https://192.168.122.152:8443/index.html": remote error: tls: handshake failure

Note that both the server and client are using rhcos-9.8.20260314-0-qemu.x86_64.qcow2, as rhcos-10.2 does not have openssl by default.

[dustymabe]

So this proves Dusty is correct, it does not matter if server side is FIPs enabled or not.

If I'm right then this PR as merged is just a duplicate test of #4373

If we decided we wanted a negative test we could either reuse the work you did to set up a separate server or just find an existing server somewhere that doesn't allow FIPs compliant algorithms.

[HuijingHei]

So this proves Dusty is correct, it does not matter if server side is FIPs enabled or not.

If I'm right then this PR as merged is just a duplicate test of #4373

If we decided we wanted a negative test we could either reuse the work you did to set up a separate server or just find an existing server somewhere that doesn't allow FIPs compliant algorithms.

I have no objection for the negative test, maybe can refer coreos.ignition.security.tls to setup 2 VMs to test.
One question: can I use fedora:43 image to build nginx container instead of rhel10/nginx-126:10.1? WDYT ?@travier
Copy Timothee's steps and change:

• Spawn a first RHCOS VM
• Have it build nginx container with non FIPS compatible TLS certs and run
• Then spawn the second FIPS VM, with ignition to fetch file from nginx server on the first VM, expect to fail