Skip to content

Conversation

@rpkelly
Copy link
Contributor

@rpkelly rpkelly commented Sep 19, 2025

Issue number:

Closes #540

Description of changes:
Adds the ability for bloodhound test overrides to exist, with reasoning, and optional messaging in the case of skipped tests. Additionally, overrides 2 tests:

  • Bottlerocket 03040101 if running on kubernetes variants will now be overridden to skip with an explanation
  • Kubernetes 04021000 was already set to automatically pass, but has been switched to use the new override system

Finally, we've removed the RSA ciphers no longer considered secure by the latest kubernetes CIS benchmark from k8s 04021200

Testing done:

Run tests:

[ssm-user@control]$ apiclient exec report cis -l 2
ctr: container "report" in namespace "default": not found
[ssm-user@control]$ apiclient report cis -l 2
Benchmark name:  CIS Bottlerocket Benchmark
Version:         v1.0.0
Reference:       https://www.cisecurity.org/benchmark/bottlerocket
Benchmark level: 2
Start time:      2025-10-20T22:08:02.530202650Z

[FAIL] 1.1.1.1   Ensure mounting of udf filesystems is disabled (Automatic)
[SKIP] 1.2.1     Ensure software update repositories are configured (Manual)
[PASS] 1.3.1     Ensure dm-verity is configured (Automatic)
[PASS] 1.4.1     Ensure setuid programs do not create core dumps (Automatic)
[PASS] 1.4.2     Ensure address space layout randomization (ASLR) is enabled (Automatic)
[PASS] 1.4.3     Ensure unprivileged eBPF is disabled (Automatic)
[PASS] 1.4.4     Ensure user namespaces are disabled (Automatic)
[PASS] 1.5.1     Ensure SELinux is configured (Automatic)
[PASS] 1.5.2     Ensure Lockdown is configured (Automatic)
[SKIP] 1.6       Ensure updates, patches, and additional security software are installed (Manual)
[PASS] 2.1.1.1   Ensure chrony is configured (Automatic)
[FAIL] 3.1.1     Ensure packet redirect sending is disabled (Automatic)
[PASS] 3.2.1     Ensure source routed packets are not accepted (Automatic)
[FAIL] 3.2.2     Ensure ICMP redirects are not accepted (Automatic)
[FAIL] 3.2.3     Ensure secure ICMP redirects are not accepted (Automatic)
[FAIL] 3.2.4     Ensure suspicious packets are logged (Automatic)
[PASS] 3.2.5     Ensure broadcast ICMP requests are ignored (Automatic)
[PASS] 3.2.6     Ensure bogus ICMP responses are ignored (Automatic)
[PASS] 3.2.7     Ensure TCP SYN Cookies is enabled (Automatic)
[FAIL] 3.3.1     Ensure SCTP is disabled (Automatic)
[SKIP] 3.4.1.1   Ensure IPv4 default deny firewall policy (Exception)
[FAIL] 3.4.1.2   Ensure IPv4 loopback traffic is configured (Automatic)
[SKIP] 3.4.1.3   Ensure IPv4 outbound and established connections are configured (Manual)
[FAIL] 3.4.2.1   Ensure IPv6 default deny firewall policy (Automatic)
[FAIL] 3.4.2.2   Ensure IPv6 loopback traffic is configured (Automatic)
[SKIP] 3.4.2.3   Ensure IPv6 outbound and established connections are configured (Manual)
[PASS] 4.1.1.1   Ensure journald is configured to write logs to persistent disk (Automatic)
[PASS] 4.1.2     Ensure permissions on journal files are configured (Automatic)

Passed:          14
Failed:          9
Skipped:         5
Total checks:    28

Compliance check result: FAIL

Exceptions:
*  3.4.1.1 Test requires iptables FORWARD chain configuration that conflicts with Kubernetes networking architecture. Kubernetes manages its own iptables rules through kube-proxy and CNI plugins.
[ssm-user@control]$ apiclient report cis-k8s -l 2
Benchmark name:  CIS Kubernetes Benchmark (Worker Node)
Version:         v1.11.1
Reference:       https://www.cisecurity.org/benchmark/kubernetes
Benchmark level: 2
Start time:      2025-10-20T22:08:10.965751661Z

[PASS] 4.1.1     Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automatic)
[PASS] 4.1.2     Ensure that the kubelet service file ownership is set to root:root (Automatic)
[SKIP] 4.1.3     If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual)
[SKIP] 4.1.4     If proxy kubeconfig file exists ensure ownership is set to root:root (Manual)
[PASS] 4.1.5     Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automatic)
[PASS] 4.1.6     Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automatic)
[PASS] 4.1.7     Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automatic)
[PASS] 4.1.8     Ensure that the client certificate authorities file ownership is set to root:root (Automatic)
[PASS] 4.1.9     If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automatic)
[PASS] 4.1.10    If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automatic)
[PASS] 4.2.1     Ensure that the --anonymous-auth argument is set to false (Automatic)
[PASS] 4.2.2     Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automatic)
[PASS] 4.2.3     Ensure that the --client-ca-file argument is set as appropriate (Automatic)
[PASS] 4.2.4     Verify that if defined, readOnlyPort is set to 0 (Automatic)
[PASS] 4.2.5     Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automatic)
[PASS] 4.2.6     Ensure that the --make-iptables-util-chains argument is set to true (Automatic)
[SKIP] 4.2.7     Ensure that the --hostname-override argument is not set (not valid for Bottlerocket) (Manual)
[SKIP] 4.2.8     Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual)
[PASS] 4.2.9     Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automatic)
[PASS] 4.2.10    Ensure that the --rotate-certificates argument is not set to false (Exception)
[PASS] 4.2.11    Verify that the RotateKubeletServerCertificate argument is set to true (Automatic)
[PASS] 4.2.12    Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automatic)
[PASS] 4.2.13    Ensure that a limit is set on pod PIDs (Automatic)
[FAIL] 4.2.14    Ensure that the --seccomp-default parameter is set to true (Automatic)
[SKIP] 4.2.15    Ensure that the --IPAddressDeny is set to any (Manual)
[SKIP] 4.3.1     Ensure that the kube-proxy metrics service is bound to localhost (Manual)

Passed:          19
Failed:          1
Skipped:         6
Total checks:    26

Compliance check result: FAIL

Exceptions:
*  4.2.10 IAM (external) auth is used on Bottlerocket, so certificate rotation does not apply. See EKS CIS Benchmark.

Check to ensure overrides are on kubernetes variants, but not ECS variants:

ec2-user /mnt   20:31  ❯ cd bottlerocket-k8s-1.33/usr/libexec/cis-checks/

ec2-user …/usr/libexec/cis-checks   20:31  ❯ ls
bottlerocket  kubernetes

ec2-user …/usr/libexec/cis-checks   20:31  ❯ ls bottlerocket/
br01010101  br01030100  br01040200  br01040400  br01050200  br02010101  br03020100  br03020300  br03020500  br03020700  br03040101       br03040102  br03040201  br03040203  br04010200
br01020100  br01040100  br01040300  br01050100  br01060000  br03010100  br03020200  br03020400  br03020600  br03030100  br03040101.json  br03040103  br03040202  br04010101  metadata.json

ec2-user …/usr/libexec/cis-checks   20:31  ❯ ls kubernetes/
k8s04010100  k8s04010300  k8s04010500  k8s04010700  k8s04010900  k8s04020100  k8s04020300  k8s04020500  k8s04020700  k8s04020900  k8s04021000.json  k8s04021200  k8s04021400  k8s04030100
k8s04010200  k8s04010400  k8s04010600  k8s04010800  k8s04011000  k8s04020200  k8s04020400  k8s04020600  k8s04020800  k8s04021000  k8s04021100       k8s04021300  k8s04021500  metadata.json

ec2-user …/usr/libexec/cis-checks   20:31  ❯ ....

ec2-user /mnt/bottlerocket-k8s-1.33   20:31  ❯ ..

ec2-user /mnt   20:31  ❯ ls
bottlerocket-ecs-2  bottlerocket-k8s-1.33  bottlerocket-root

ec2-user /mnt   20:31  ❯ cd bottlerocket-ecs-2/usr/libexec/cis-checks/

ec2-user …/usr/libexec/cis-checks   20:31  ❯ ls
bottlerocket

ec2-user …/usr/libexec/cis-checks   20:31  ❯ ls bottlerocket/
br01010101  br01030100  br01040200  br01040400  br01050200  br02010101  br03020100  br03020300  br03020500  br03020700  br03040101  br03040103  br03040202  br04010101  metadata.json
br01020100  br01040100  br01040300  br01050100  br01060000  br03010100  br03020200  br03020400  br03020600  br03030100  br03040102  br03040201  br03040203  br04010200

json output sample:

   "error": ""
    },
    "br03030100": {
      "name": "br03030100",
      "id": "3.3.1",
      "level": 2,
      "title": "Ensure SCTP is disabled",
      "mode": "Automatic",
      "status": "FAIL",
      "error": "modprobe for sctp is not disabled"
    },
    "br03040101": {
      "name": "br03040101",
      "id": "3.4.1.1",
      "level": 2,
      "title": "Ensure IPv4 default deny firewall policy",
      "mode": "Automatic",
      "status": "SKIP",
      "error": "Test requires iptables FORWARD chain configuration that conflicts with Kubernetes networking architecture. Kubernetes manages its own iptables rules through kube-proxy and CNI plugins."
    },
    "br03040102": {
      "name": "br03040102",
      "id": "3.4.1.2",
      "level": 2,
      "title": "Ensure IPv4 loopback traffic is configured",
      "mode": "Automatic",
      "status": "FAIL",
      "error": "Unable to find expected iptables INPUT values"
    },
    "br03040103": {
      "name": "br03040103",
      "id": "3.4.1.3",

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@rpkelly rpkelly force-pushed the bloodhound-changes branch 5 times, most recently from 66e17a2 to cc2e11b Compare September 19, 2025 23:44
@ytsssun
Copy link
Contributor

ytsssun commented Sep 22, 2025

Q - could you explain on the failed tests in apiclient report -l 2 cis and are they expected?

@rpkelly
Copy link
Contributor Author

rpkelly commented Sep 29, 2025

@ytsssun The fails are expected. I did not set the image to be L2 CIS compliant, I only wanted to validate that the overrides were being properly read.

@bcressey
Copy link
Contributor

The formatting for this line is wonky in plain text output:

[SKIPPED: Test requires iptables FORWARD chain configuration that conflicts with Kubernetes networking architecture. Kubernetes manages its own iptables rules through kube-proxy and CNI plugins.] 3.4.1.1   Ensure IPv4 default deny firewall policy (Automatic)

Ideally we'd format it more like:

[SKIP] 3.4.1.1   Ensure IPv4 default deny firewall policy (Exception)

Exceptions:
*  3.4.1.1 requires iptables FORWARD chain configuration that conflicts with Kubernetes networking architecture. Kubernetes manages its own iptables rules through kube-proxy and CNI plugins.

@rpkelly
Copy link
Contributor Author

rpkelly commented Sep 30, 2025

Added updated code, will rebuild and run a test image to show the new output next

@rpkelly rpkelly force-pushed the bloodhound-changes branch from 0dd21ce to c34ff53 Compare October 1, 2025 18:15
@rpkelly
Copy link
Contributor Author

rpkelly commented Oct 8, 2025

Updated test output has been added

Copy link
Contributor

@bcressey bcressey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very close, but it seems like it'd be nice if the implementation respected the status value provided in the override file when rendering the report.

Comment on lines 1 to 5
{
"reason": "Test requires iptables FORWARD chain configuration that conflicts with Kubernetes networking architecture. Kubernetes manages its own iptables rules through kube-proxy and CNI plugins.",
"reference": "https://github.com/bottlerocket-os/bottlerocket/issues/540",
"status": "SKIP"
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need an override for the IPv6 check, 3.4.2.1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per #540 3.4.2.1 actually does pass if properly configured, as we only seem to override the IPv4 rules.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per #540 3.4.2.1 actually does pass if properly configured, as we only seem to override the IPv4 rules.

But does the node actually work properly if using IPv6 for the overlay network, with forwarding disabled? I'm not sure why that would be true for IPv6 but not true for IPv4.

@rpkelly rpkelly force-pushed the bloodhound-changes branch from c34ff53 to 0ade908 Compare October 17, 2025 22:32
Add override system to skip specific compliance tests through JSON
configuration files. Enables variants to skip incompatible
tests while maintaining compliance checking.

Signed-off-by: Richard Kelly <[email protected]>
Replace ManualChecker with actual K8S04021000Checker implementation
and add override to pass on Bottlerocket.

Signed-off-by: Richard Kelly <[email protected]>
Remove TLS_RSA_WITH_AES_128_GCM_SHA256 and TLS_RSA_WITH_AES_256_GCM_SHA384
from allowed cipher suites per latest Kubernetes CIS benchmark.

Signed-off-by: Richard Kelly <[email protected]>
@rpkelly rpkelly force-pushed the bloodhound-changes branch from 0ade908 to cba0a94 Compare October 17, 2025 23:04
Copy link
Contributor

@bcressey bcressey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are exceptions shown in the JSON output?

Comment on lines +413 to +417
%package -n %{_cross_os}bloodhound-k8s-overrides
Summary: CIS Test overrides for Kubernetes variants
Requires: (%{_cross_os}bloodhound and %{_cross_os}variant-runtime(k8s))
%description -n %{_cross_os}bloodhound-k8s-overrides
%{summary}.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can just add these into the existing bloodhound-k8s subpackage which has the same requirements

The added test overrides do not directly flatten into the previous json
output structure. In order to maintain the same json structure we
require a custom serializer

Signed-off-by: Richard Kelly <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CIS Benchmark Level 2 benchmark failure on EKS

3 participants