OCPBUGS-18027: explain use of multiple AWS security groups

[jeana-redhat]

Version(s):
4.12-4.15

Issue:
OCPBUGS-18027

Link to docs preview:
Sample YAML for a compute machine set custom resource on AWS:

• infrastructure
• standard
• edge zones

QE review:

• QE has approved this change.

Additional information:
Also fixes mixed up callouts

[openshift-ci-robot]

@jeana-redhat: This pull request references Jira Issue OCPBUGS-18027, which is invalid:

• expected Jira Issue OCPBUGS-18027 to depend on a bug in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Version(s):
4.12-4.15

Issue:
OCPBUGS-18027

Link to docs preview:

QE review:

• QE has approved this change.

Additional information:
Also fixes mixed up callouts

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jeana-redhat]

@damdo do you remember https://issues.redhat.com/browse/OCPBUGS-11524 from 2023? trying to get some of these bugs outta here this sprint 😅

(https://issues.redhat.com/browse/OCPBUGS-18027 is the corresponding docs bug)

Edit: something is messed up with the edge annotations here and is making the build fail. But the new content should be reviewable and I can fix the numbering tomorrow :)

[jeana-redhat]

I moved some other things around to fix numbering, but this one is the new content

[jeana-redhat]

New sample YAML

[ocpdocs-previewbot]

🤖 Wed Mar 05 14:57:01 - Prow CI generated the docs preview:

https://89569--ocpdocs-pr.netlify.app/openshift-enterprise/latest/machine_management/creating-infrastructure-machinesets.html
https://89569--ocpdocs-pr.netlify.app/openshift-enterprise/latest/machine_management/creating_machinesets/creating-machineset-aws.html
https://89569--ocpdocs-pr.netlify.app/openshift-enterprise/latest/post_installation_configuration/aws-compute-edge-zone-tasks.html

[damdo]

Hey @jeana-redhat thanks for putting this together 👍

I had a look at the bug you linked.
Wouldn't this yield scenario 2 from the testing @sunzhaohua2 conducted here ?

If there are multiple groups and both have the kubernetes.io/cluster/<infra-id> tag, this is considered an error state and the CCM will not reconcile the security group rules

@sunzhaohua2 could you provide guidance on this?

[jeana-redhat]

Hey @jeana-redhat thanks for putting this together 👍

I had a look at the bug you linked. Wouldn't this yield scenario 2 from the testing @sunzhaohua2 conducted here ?

If there are multiple groups and both have the kubernetes.io/cluster/<infra-id> tag, this is considered an error state and the CCM will not reconcile the security group rules

@sunzhaohua2 could you provide guidance on this?

Yeah, I am probably not understanding where that tag plays in. I thought by having the value <optional_security_group> different than <infrastructure_id>-worker-sg, that got around the issue, but I likely am just misunderstanding something about the config here :)

I see kubernetes.io/cluster/<infrastructure_id> is down in providerSpec.value.tags, but it's still impacting the SG setting I guess? Snipped out spec bits:

providerSpec:
        value:
          [...]
          securityGroups:
            - filters:
                - name: tag:Name
                  values:
                    - <infrastructure_id>-worker-sg
            - filters: 
                - name: tag:Name
                  values:
                    - <optional_security_group>
          [...]
          tags:
            - name: kubernetes.io/cluster/<infrastructure_id>
              value: owned
            - name: <custom_tag_name> 
              value: <custom_tag_value>

So do I need to somehow relate these things to each other in a way that only one SG gets the kubernetes.io/cluster/<infrastructure_id> tag? Not sure how exactly to make that happen 🤔

[jeana-redhat]

I moved the callout for tags to the top level to make it easier to talk about this section more generally

[jeana-redhat]

I changed this from

Optional: Specify custom tag data for your cluster.

to

Optional: Specify custom tag data for your cluster in addition to the existing `kubernetes.io/cluster/<infrastructure_id>` tag.

because I moved the callout from the custom tag name line to the tags stanza

[jeana-redhat]

Refreshed the PR with fixes to the callouts so that it will build correctly

[openshift-ci]

@jeana-redhat: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[sunzhaohua2]

@jeana-redhat @damdo I will check this tomorrow

[jeana-redhat]

@jeana-redhat @damdo I will check this tomorrow

Thanks! No rush :)

[sunzhaohua2]

Hi @jeana-redhat I checked this bug and I tested again on a 4.15 cluster, I can confirm Joel mentioned here are correct.
When you have multiple groups:

• If there are multiple groups and neither has the Kubernetes.io/cluster/<infra-id> tag, this is considered an error state and the CCM will not reconcile the security group rules
• If there are multiple groups and both have the Kubernetes.io/cluster/<infra-id> tag, this is considered an error state and the CCM will not reconcile the security group rules
• If there are multiple groups and a single group has the Kubernetes.io/cluster/<infra-id> tag, this is considered correct and the tagged group is reconciled with the security group rules

But regarding the traffic issue mentioned by Dam, I am not sure whether we should mention that multiple SGs are supported in the doc, even though the machine creation was successful and CCM did not report any errors. @damdo, do you have any concerns about this?

And for the new versions, by default there are 2 SGs with tags in machineset, the tag is not Kubernetes.io/cluster/<infra-id> , is sigs.k8s.io/cluster-api-provider-aws/cluster/<infra-id>, I am not sure whether the conclusion mentioned by Joel above is still valid. I need some testing.

4.19.0-0.nightly-2025-03-05-160850 by default：

$ oc edit machineset zhsun-aws37-nrfd5-worker-us-east-2c
          securityGroups:
          - filters:
            - name: tag:Name
              values:
              - zhsun-aws37-nrfd5-node
          - filters:
            - name: tag:Name
              values:
              - zhsun-aws37-nrfd5-lb

for 4.15.0-0.nightly-2025-03-05-202732 by default：

$ oc edit machineset zhsun-aws15-5xnl9-worker-us-east-2b
          securityGroups:
          - filters:
            - name: tag:Name
              values:
              - zhsun-aws15-5xnl9-worker-sg

[jeana-redhat]

Hey, thank you!

Can you post a spec that shows where these tags are?

And for the new versions, by default there are 2 SGs with tags in machineset, the tag is not Kubernetes.io/cluster/<infra-id> , is sigs.k8s.io/cluster-api-provider-aws/cluster/<infra-id>, I am not sure whether the conclusion mentioned by Joel above is still valid. I need some testing.

I ask because I see one in providerSpec.value.tags in the sample spec, but I am not sure how it relates to the SG fields.

[sunzhaohua2]

Hey, thank you!

Can you post a spec that shows where these tags are?

And for the new versions, by default there are 2 SGs with tags in machineset, the tag is not Kubernetes.io/cluster/<infra-id> , is sigs.k8s.io/cluster-api-provider-aws/cluster/<infra-id>, I am not sure whether the conclusion mentioned by Joel above is still valid. I need some testing.

I ask because I see one in providerSpec.value.tags in the sample spec, but I am not sure how it relates to the SG fields.

Hi @jeana-redhat I was checked from aws console, for the 2 SGs in machineset, I checked again zhsun-aws37-nrfd5-lb is lb sg, it has tag Kubernetes.io/cluster/<infra-id>

last week I checked the wrong sg zhsun-aws37-nrfd5-node, it doesn't have tag Kubernetes.io/cluster/<infra-id>

And I tested again on 4.19, the results are same with before. For this issue If there are multiple groups and both have the kubernetes.io/cluster/<infra-id> tag, this is considered an error state and the CCM will not reconcile the security group rules maybe we need to add a note to remind users not to add tag Kubernetes.io/cluster/<infra-id> when creating SG.