Refactor authentication mechanism from SCRAM to SASL/PLAIN and update related documentation and configurations

Author: hguerrero

README.md

@@ -3,7 +3,7 @@
 A runnable demo of **developer self-service for Apache Kafka**. Application teams
 discover available topics in Backstage (published as AsyncAPI), then use a scaffolder
 template to request access. Behind the form, Kong Event Gateway gets a dedicated
-**virtual cluster** for the app, **SCRAM (or OAuth) credentials**, and **ACLs** scoped
+**virtual cluster** for the app, **credentials (SASL username + password, or OAuth)**, and **ACLs** scoped
 to exactly the topics requested — delivered via GitOps and reconciled by the **Kong
 Operator** on Kubernetes.
 
@@ -43,7 +43,7 @@ Operator** on Kubernetes.
 │   ├── backstage/           #   in-cluster Backstage: Deployment, Service, Postgres, config
 │   ├── argocd/              #   Argo CD route through the Kong Gateway (+ insecure mode)
 │   └── kafka/               #   Strimzi Kafka cluster + topics
-├── examples/kafka-client/   # SCRAM client config + test commands
+├── examples/kafka-client/   # SASL/PLAIN client config + test commands
 ├── scripts/                 # bootstrap, cert, validate
 └── docs/                    # architecture, flows, prerequisites, secrets, repositories
 ```
@@ -131,9 +131,9 @@ repo's `catalog-info.yaml` (see `platform/backstage/app-config.configmap.yaml`).
 
 1. A developer opens the **Retail Banking NY** or **Wealth Management LA** API in the
    Backstage catalog and reads the AsyncAPI channels (topics).
-2. They run **Consume Kafka Topics**, name their app, pick topics, and choose SCRAM or
-   OAuth. Backstage opens a PR adding `apps/<app>/` to the `kafka-selfservice-gitops`
-   repo.
+2. They run **Consume Kafka Topics**, name their app, pick topics, and choose
+   SASL/PLAIN or OAuth. Backstage opens a PR adding `apps/<app>/` to the
+   `kafka-selfservice-gitops` repo.
 3. On merge, Argo CD + Kong Operator provision the virtual cluster, credentials, ACLs
    and route. The app connects to `bootstrap.<app>.127-0-0-1.sslip.io:9092`.
 4. Need more topics later? **Add Topics to Application** changes only the ACL policy.
@@ -149,14 +149,18 @@ See [`docs/flows.md`](docs/flows.md) for sequence diagrams and
 
 ## Notes & caveats
 
-- Kong Operator CRD field names for Event Gateway are evolving. The virtual cluster
-  `apiSpec` mirrors the Konnect / `kongctl` schema. Two spots to confirm against the
-  CRDs in *your* operator version: the SCRAM principal **password secret-ref** shape,
-  and whether ACLs are a separate `EventGatewayVirtualClusterPolicy` or inline on the
-  virtual cluster (`spec.apiSpec.clusterPolicies`). Both are called out in comments.
-- SCRAM passwords (and the Backstage/Konnect secrets) are placeholders (`REPLACE_ME`).
-  For real use, generate them and store via Sealed Secrets or External Secrets — never
-  commit plaintext. [`docs/secrets.md`](docs/secrets.md) shows worked manifests for both.
+- Auth mechanism: the Kong Operator CRD's `saslScram` type does **not** accept inline
+  username/password (it only carries `algorithm` and resolves principals via Kong
+  Identity). So the self-contained "issue a user + password, validated and terminated
+  at the gateway" model uses **`saslPlain`** (`type: saslPlain` + a sibling `saslPlain`
+  object with `mediation: terminate` and `principals[]`). Native SCRAM or OAuth via
+  Kong Identity is the enterprise path. Auth and ACL manifests match the installed CRD
+  schema (`configuration.konghq.com/v1alpha1`): auth is a discriminated union, and ACL
+  rules use `resourceType` / `operations: [{name}]` / `resourceNames: {type: stat, stat: [{match}]}`.
+- Credentials (and the Backstage/Konnect secrets) are placeholders (`REPLACE_ME`).
+  For real use, use a secret-template / Konnect vault reference for the SASL password,
+  and store Kubernetes secrets via Sealed Secrets or External Secrets — never commit
+  plaintext. [`docs/secrets.md`](docs/secrets.md) shows worked manifests.
 - The `platform/` Kafka + Event Gateway manifests are adapted from the
   `kong-event-gw-kubernetes` reference.
 

backstage/templates/add-topics-to-app/skeleton/kong/acl-policy.yaml

@@ -25,18 +25,22 @@ spec:
       config:
         rules:
           - action: allow
-            resource_type: topic
+            resourceType: topic
             operations:
               - name: describe
               - name: read
-            resource_names:
-            {%- for topic in values.topics %}
-              - match: "${{ topic }}"
-            {%- endfor %}
+            resourceNames:
+              type: stat
+              stat:
+              {%- for topic in values.topics %}
+                - match: "${{ topic }}"
+              {%- endfor %}
           - action: allow
-            resource_type: group
+            resourceType: group
             operations:
               - name: describe
               - name: read
-            resource_names:
-              - match: "${{ values.appName }}.*"
+            resourceNames:
+              type: stat
+              stat:
+                - match: "${{ values.appName }}.*"

backstage/templates/consume-kafka-topics/skeleton/README.md

@@ -16,10 +16,9 @@ Self-service Kafka access for **${{ values.appName }}**, generated by Backstage.
 
 | File | Purpose |
 |------|---------|
-| `kong/virtual-cluster.yaml` | Dedicated virtual cluster + authentication + prefix hiding |
+| `kong/virtual-cluster.yaml` | Dedicated virtual cluster + authentication (credentials terminated at the gateway) + prefix hiding |
 | `kong/acl-policy.yaml` | Read ACLs for the topics above — **edit this to change access** |
 | `kong/tlsroute.yaml` | SNI route so clients reach this virtual cluster |
-| `kong/scram-credentials.yaml` | SCRAM secret (SCRAM auth only) |
 | `catalog-info.yaml` | Backstage Component for this app |
 
 On merge, Argo CD applies `kong/` and Kong Operator reconciles the changes into the

backstage/templates/consume-kafka-topics/skeleton/kong/acl-policy.yaml

@@ -4,15 +4,13 @@
 # topics plus its own consumer groups. Everything else is denied because the virtual
 # cluster runs in enforce_on_gateway mode.
 #
-# THIS IS THE FILE YOU EDIT LATER. To grant access to more topics, add them to the
-# "Add topics" self-service template (or append resource_names below) - nothing else
-# about the application changes.
+# THIS IS THE FILE YOU EDIT LATER. To grant more topics, re-run "Add Topics to
+# Application" (or add entries under resourceNames.stat below) - nothing else about
+# the application changes.
 #
-# NOTE on the CRD kind: this follows the same reference pattern as the
-# EventGatewayVirtualClusterConsumePolicy / ...ProducePolicy CRDs. Depending on your
-# Kong Operator version, ACLs may instead be expressed inline on the
-# EventGatewayVirtualCluster (spec.apiSpec.clusterPolicies) using the identical
-# config below. Verify against your installed CRDs and move the block if needed.
+# CRD schema: rule fields are camelCase (`resourceType`, `resourceNames`);
+# `operations` is a list of `{name: <op>}`; `resourceNames` is one object with a
+# `type: stat` discriminator whose `stat` is a list of `{match: <glob>}`.
 apiVersion: configuration.konghq.com/v1alpha1
 kind: EventGatewayVirtualClusterPolicy
 metadata:
@@ -36,19 +34,23 @@ spec:
       config:
         rules:
           - action: allow
-            resource_type: topic
+            resourceType: topic
             operations:
               - name: describe
               - name: read
-            resource_names:
-            {%- for topic in values.topics %}
-              - match: "${{ topic }}"
-            {%- endfor %}
+            resourceNames:
+              type: stat
+              stat:
+              {%- for topic in values.topics %}
+                - match: "${{ topic }}"
+              {%- endfor %}
           - action: allow
-            resource_type: group
+            resourceType: group
             operations:
               - name: describe
               - name: read
-            resource_names:
-              # This app may only use consumer groups it prefixes with its own name.
-              - match: "${{ values.appName }}.*"
+            resourceNames:
+              type: stat
+              stat:
+                # This app may only use consumer groups it prefixes with its own name.
+                - match: "${{ values.appName }}.*"

backstage/templates/consume-kafka-topics/skeleton/kong/kustomization.yaml

@@ -9,4 +9,3 @@ resources:
   - virtual-cluster.yaml
   - acl-policy.yaml
   - tlsroute.yaml
-  - scram-credentials.yaml

backstage/templates/consume-kafka-topics/skeleton/kong/scram-credentials.yaml

@@ -5,6 +5,8 @@
 # - the "${{ values.namespacePrefix }}" broker prefix is hidden from the client
 # - aclMode: enforce_on_gateway -> principals start with NO access; the ACL policy
 #   in acl-policy.yaml grants read on exactly the requested topics.
+#
+# Auth is a discriminated union: `type: <x>` + a sibling object named `<x>`.
 apiVersion: configuration.konghq.com/v1alpha1
 kind: EventGatewayVirtualCluster
 metadata:
@@ -26,33 +28,32 @@ spec:
       prefix: "${{ values.namespacePrefix }}"
       mode: hide_prefix
     authentication:
-    {%- if values.authMode == "scram" %}
-      # SASL/SCRAM-SHA-256, terminated at the gateway: the credential is validated
-      # here and never forwarded to Kafka. The password lives in the Secret rendered
-      # in scram-credentials.yaml.
-      # NOTE: the exact password secret-ref shape is operator-version dependent -
-      # confirm against your installed EventGatewayVirtualCluster CRD. If your
-      # version only accepts inline principals, copy the password value in directly.
-      - type: sasl_scram
-        mediation: terminate
-        mechanism: SCRAM-SHA-256
-        principals:
-          - username: ${{ values.appName }}
-            password:
-              type: secretRef
-              secretRef:
-                name: ${{ values.appName }}-kafka-scram
-                key: password
-    {%- else %}
+    {%- if values.authMode == "oauth" %}
       # OAuth / OIDC: client presents a bearer token; the gateway verifies it against
       # the configured auth server (e.g. Kong Identity). The ACL condition in
-      # acl-policy.yaml matches on the token's principal/subject claim.
+      # acl-policy.yaml matches on the token principal/subject claim.
       # See: https://developer.konghq.com/how-to/event-gateway/kong-identity-oauth/
-      - type: oauth
-        mediation: terminate
-        oauth:
-          # Replace with your issuer / JWKS endpoint (Kong Identity or external IdP).
-          issuer: https://<your-kong-identity-issuer>
-          # The subject/client id that identifies this application in tokens.
-          audience: ${{ values.appName }}
+      - type: oauthBearer
+        oauthBearer:
+          mediation: terminate
+          jwks:
+            # Replace with your issuer's JWKS endpoint (Kong Identity or external IdP).
+            endpoint: https://<your-issuer>/.well-known/jwks.json
+          validate:
+            issuer: https://<your-issuer>
+            audiences:
+              - name: ${{ values.appName }}
+    {%- else %}
+      # SASL/PLAIN, terminated at the gateway: the username + password are validated
+      # here and never forwarded to Kafka. (The CRD's saslScram type does not accept
+      # inline credentials - it needs Kong Identity - so saslPlain is used for the
+      # self-contained "issue a user + password" model.)
+      - type: saslPlain
+        saslPlain:
+          mediation: terminate
+          principals:
+            # Demo uses an inline password. In production use a secret-template
+            # expression / Konnect vault reference instead of a literal value.
+            - username: ${{ values.appName }}
+              password: "REPLACE_ME"
     {%- endif %}

backstage/templates/consume-kafka-topics/skeleton/kong/virtual-cluster.yaml

@@ -7,9 +7,10 @@ metadata:
   description: |
     Request self-service access to Northwind Financial Kafka topics through Kong
     Event Gateway. Provisions a dedicated virtual cluster for your application,
-    issues SCRAM (or OAuth) credentials, and creates ACLs scoped to exactly the
-    topics you select. Changes are delivered as a GitOps pull request; on merge,
-    Kong Operator reconciles the control plane, data plane and access rules.
+    issues credentials (SASL/PLAIN username + password, or OAuth), and creates ACLs
+    scoped to exactly the topics you select. Changes are delivered as a GitOps pull
+    request; on merge, Kong Operator reconciles the control plane, data plane and
+    access rules.
   tags:
     - kafka
     - event-gateway
@@ -60,10 +61,10 @@ spec:
         authMode:
           title: Authentication
           type: string
-          default: scram
-          enum: [scram, oauth]
+          default: plain
+          enum: [plain, oauth]
           enumNames:
-            - SASL/SCRAM-SHA-256 (username + password)
+            - SASL/PLAIN (username + password, terminated at the gateway)
             - OAuth / OIDC (bearer token via Kong Identity)
       dependencies:
         businessUnit:

backstage/templates/consume-kafka-topics/template.yaml

@@ -9,7 +9,7 @@ info:
     below are exactly what clients use.
 
     To consume any of these channels, use the **Consume Kafka Topics** self-service
-    template in Backstage. It provisions a virtual cluster, SCRAM (or OAuth)
+    template in Backstage. It provisions a virtual cluster, SASL (or OAuth)
     credentials, and ACLs scoped to the channels you select.
   contact:
     name: Retail Banking NY Platform Team
@@ -29,7 +29,7 @@ servers:
         description: Your provisioned application (virtual cluster) name.
         default: your-app
     security:
-      - $ref: "#/components/securitySchemes/scram"
+      - $ref: "#/components/securitySchemes/saslPlain"
     bindings:
       kafka:
         schemaRegistryUrl: ""
@@ -76,9 +76,9 @@ operations:
       $ref: "#/channels/core.accounts.status.v2"
 components:
   securitySchemes:
-    scram:
-      type: scramSha256
-      description: SASL/SCRAM-SHA-256 over TLS. Credentials are issued per application.
+    saslPlain:
+      type: plain
+      description: SASL/PLAIN over TLS. Credentials are issued per application and terminated at the gateway.
   messages:
     CardDispatched:
       name: CardDispatched

catalog/specs/retail-banking-ny.asyncapi.yaml

@@ -26,7 +26,7 @@ servers:
         description: Your provisioned application (virtual cluster) name.
         default: your-app
     security:
-      - $ref: "#/components/securitySchemes/scram"
+      - $ref: "#/components/securitySchemes/saslPlain"
 channels:
   advisor.daily-client-activity.v1:
     address: advisor.daily-client-activity.v1
@@ -72,9 +72,9 @@ operations:
       $ref: "#/channels/infosec.security.fraud.risk-scores.v3"
 components:
   securitySchemes:
-    scram:
-      type: scramSha256
-      description: SASL/SCRAM-SHA-256 over TLS. Credentials are issued per application.
+    saslPlain:
+      type: plain
+      description: SASL/PLAIN over TLS. Credentials are issued per application and terminated at the gateway.
   messages:
     AdvisorActivity:
       name: AdvisorActivity