Refactor GitOps structure and documentation for clarity; remove deprecated files and update references to the kafka-selfservice-gitops repository.

Author: hguerrero

.github/workflows/validate.yaml

@@ -4,7 +4,6 @@ on:
   pull_request:
     paths:
       - "platform/**"
-      - "gitops/**"
       - "catalog/**"
       - "backstage/**"
   push:
@@ -21,16 +20,15 @@ jobs:
           pip install yamllint
           # Skeletons contain Nunjucks (${{ ... }}) and are not valid YAML on their own.
           yamllint -d relaxed \
-            $(git ls-files 'platform/**/*.yaml' 'gitops/**/*.yaml' 'catalog/**/*.yaml') || true
+            $(git ls-files 'platform/**/*.yaml' 'catalog/**/*.yaml') || true
 
       - name: Validate rendered manifests against Kubernetes + CRD schemas
         run: |
           curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
           # -ignore-missing-schemas because Kong Operator / Strimzi CRDs aren't in the
           # default schema set. For strict CRD validation, add -schema-location pointing
           # at your extracted CRD OpenAPI schemas.
-          ./kubeconform -ignore-missing-schemas -summary \
-            platform gitops/apps
+          ./kubeconform -ignore-missing-schemas -summary platform
 
   asyncapi:
     runs-on: ubuntu-latest

README.md

@@ -10,6 +10,12 @@ Operator** on Kubernetes.
 > The first request delivers everything (control plane, data plane, credentials,
 > ACLs, routing). Later requests usually just update the ACLs to add more topics.
 
+> **Two repos.** This is the **portal/platform** repo (catalog, templates, platform
+> bootstrap). Tenant GitOps config lives in a companion repo,
+> [`kafka-selfservice-gitops`](https://github.com/your-org/kafka-selfservice-gitops):
+> the templates open PRs there and Argo CD watches it. See
+> [`docs/repositories.md`](docs/repositories.md).
+
 ## Why it's interesting
 
 - **Zero broker changes to grant access.** Credentials are terminated at the gateway
@@ -36,15 +42,14 @@ Operator** on Kubernetes.
 │   ├── networking/          #   GatewayClass/Config, Gateway, TLS cert
 │   ├── backstage/           #   in-cluster Backstage: Deployment, Service, Postgres, config
 │   └── kafka/               #   Strimzi Kafka cluster + topics
-├── gitops/
-│   ├── argocd/              # platform App + tenants ApplicationSet
-│   └── apps/                # one dir per onboarded app (Backstage writes here)
-│       └── fraud-analytics/ #   worked example (rendered output)
 ├── examples/kafka-client/   # SCRAM client config + test commands
 ├── scripts/                 # bootstrap, cert, validate
-└── docs/                    # architecture.md, flows.md
+└── docs/                    # architecture, flows, prerequisites, secrets, repositories
 ```
 
+Tenant config (the `apps/*` directories Backstage writes to, plus the Argo CD
+`ApplicationSet`) lives in the companion **`kafka-selfservice-gitops`** repo, not here.
+
 ## Prerequisites
 
 - A Kubernetes cluster (minikube is fine) with **Kong Operator**, **Strimzi**,
@@ -64,12 +69,15 @@ The essentials:
 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
 ```
 
-**Argo CD** — reconciles `platform/` and auto-onboards each `gitops/apps/*` tenant:
+**Argo CD** — reconciles `platform/` (this repo) and auto-onboards each tenant from
+the companion repo's `apps/*`:
 
 ```bash
 kubectl create namespace argocd
 kubectl apply -n argocd --server-side --force-conflicts \
   -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
+# then apply argocd/platform-app.yaml + argocd/tenants-appset.yaml from
+# the kafka-selfservice-gitops repo (edit repoURLs first).
 ```
 
 **Backstage** — runs in-cluster; build the image once, then deploy the manifests:
@@ -100,25 +108,27 @@ Kong Operator + Strimzi install commands (and the AsyncAPI page wiring) are also
 export KONNECT_PAT=kpat_xxx
 ./scripts/bootstrap.sh
 
-# 2) Either let Argo CD manage tenants...
-kubectl apply -f gitops/argocd/            # edit repoURL first
-# ...or apply the worked example directly:
-kubectl apply -k gitops/apps/fraud-analytics/kong/
+# 2) Tenants come from the companion repo (kafka-selfservice-gitops):
+#    let Argo CD manage them...
+kubectl apply -f ../kafka-selfservice-gitops/argocd/   # edit repoURLs first
+#    ...or apply the worked example directly:
+kubectl apply -k ../kafka-selfservice-gitops/apps/fraud-analytics/kong/
 
 # 3) Expose the gateway locally and test
 minikube tunnel &
 cat examples/kafka-client/test-commands.md
 ```
 
-To wire up the portal, register `catalog-info.yaml` in Backstage and merge
-`backstage/app-config.snippet.yaml` into your `app-config.yaml`.
+To wire up the portal, point the in-cluster Backstage `catalog.locations` at this
+repo's `catalog-info.yaml` (see `platform/backstage/app-config.configmap.yaml`).
 
 ## The self-service experience
 
 1. A developer opens the **Retail Banking NY** or **Wealth Management LA** API in the
    Backstage catalog and reads the AsyncAPI channels (topics).
 2. They run **Consume Kafka Topics**, name their app, pick topics, and choose SCRAM or
-   OAuth. Backstage opens a PR under `gitops/apps/<app>/`.
+   OAuth. Backstage opens a PR adding `apps/<app>/` to the `kafka-selfservice-gitops`
+   repo.
 3. On merge, Argo CD + Kong Operator provision the virtual cluster, credentials, ACLs
    and route. The app connects to `bootstrap.<app>.127-0-0-1.sslip.io:9092`.
 4. Need more topics later? **Add Topics to Application** changes only the ACL policy.
@@ -139,9 +149,9 @@ See [`docs/flows.md`](docs/flows.md) for sequence diagrams and
   CRDs in *your* operator version: the SCRAM principal **password secret-ref** shape,
   and whether ACLs are a separate `EventGatewayVirtualClusterPolicy` or inline on the
   virtual cluster (`spec.apiSpec.clusterPolicies`). Both are called out in comments.
-- SCRAM passwords are placeholders (`REPLACE_ME`). For real use, generate them with a
-  custom scaffolder action and store via SealedSecrets / External Secrets — never
-  commit plaintext.
+- SCRAM passwords (and the Backstage/Konnect secrets) are placeholders (`REPLACE_ME`).
+  For real use, generate them and store via Sealed Secrets or External Secrets — never
+  commit plaintext. [`docs/secrets.md`](docs/secrets.md) shows worked manifests for both.
 - The `platform/` Kafka + Event Gateway manifests are adapted from the
   `kong-event-gw-kubernetes` reference.
 

backstage/templates/add-topics-to-app/template.yaml

@@ -76,8 +76,9 @@ spec:
       required: [repoUrl]
       properties:
         repoUrl:
-          title: GitOps repository
+          title: GitOps config repository
           type: string
+          description: The kafka-selfservice-gitops repo (Argo CD watches its apps/*).
           ui:field: RepoUrlPicker
           ui:options:
             allowedHosts:
@@ -89,7 +90,7 @@ spec:
       action: fetch:template
       input:
         url: ./skeleton
-        targetPath: gitops/apps/${{ parameters.appName }}
+        targetPath: apps/${{ parameters.appName }}
         values:
           appName: ${{ parameters.appName }}
           businessUnit: ${{ parameters.businessUnit }}

backstage/templates/consume-kafka-topics/template.yaml

@@ -105,9 +105,11 @@ spec:
       required: [repoUrl]
       properties:
         repoUrl:
-          title: GitOps repository
+          title: GitOps config repository
           type: string
-          description: Repository that Argo CD watches for tenant manifests.
+          description: >
+            The kafka-selfservice-gitops repo (Argo CD watches its apps/*). PRs from
+            this template land there, not in the portal/platform repo.
           ui:field: RepoUrlPicker
           ui:options:
             allowedHosts:
@@ -119,7 +121,7 @@ spec:
       action: fetch:template
       input:
         url: ./skeleton
-        targetPath: gitops/apps/${{ parameters.appName }}
+        targetPath: apps/${{ parameters.appName }}
         values:
           appName: ${{ parameters.appName }}
           owner: ${{ parameters.owner }}
@@ -155,7 +157,7 @@ spec:
       action: catalog:register
       input:
         repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
-        catalogInfoPath: /gitops/apps/${{ parameters.appName }}/catalog-info.yaml
+        catalogInfoPath: /apps/${{ parameters.appName }}/catalog-info.yaml
         optional: true
 
   output:

docs/architecture.md

@@ -5,8 +5,8 @@
 | Layer | What it is | In this repo |
 |-------|------------|--------------|
 | Developer portal | Backstage (in-cluster) — catalog of streams (AsyncAPI) + self-service templates | `catalog/`, `backstage/`, deployed via `platform/backstage/` |
-| Delivery | Git + Argo CD (ApplicationSet per tenant) | `gitops/` |
-| Control | Kong Operator reconciling Konnect + data plane from CRDs | `platform/`, `gitops/apps/*/kong/` |
+| Delivery | Git + Argo CD (ApplicationSet per tenant) | `kafka-selfservice-gitops` repo |
+| Control | Kong Operator reconciling Konnect + data plane from CRDs | `platform/`, config repo `apps/*/kong/` |
 | Data | Kong Event Gateway data plane (KNEP) fronting Kafka | `platform/event-gateway/`, `platform/networking/` |
 | Backing store | Strimzi-managed Apache Kafka (`northwind`) | `platform/kafka/` |
 
@@ -15,7 +15,7 @@
 ```mermaid
 flowchart LR
   dev([App developer]) -->|browse AsyncAPI, pick topics| bs[Backstage template]
-  bs -->|opens PR: gitops/apps/<app>/| git[(Git repo)]
+  bs -->|opens PR: apps/<app>/| git[(config repo)]
   git -->|ApplicationSet syncs| argo[Argo CD]
   argo -->|applies CRDs| ko[Kong Operator]
   ko -->|configures| konnect[(Konnect control plane)]

docs/flows.md

@@ -18,7 +18,7 @@ sequenceDiagram
   participant KO as Kong Operator
   participant KEG as Event Gateway
   Dev->>BS: Pick provider + topics + auth (SCRAM)
-  BS->>Git: PR adds gitops/apps/<app>/ (VC, ACL, TLSRoute, Secret)
+  BS->>Git: PR adds apps/<app>/ to config repo (VC, ACL, TLSRoute, Secret)
   Dev->>Git: Review + merge
   Git->>Argo: ApplicationSet detects new tenant dir
   Argo->>KO: Apply CRDs
@@ -36,7 +36,7 @@ What "deliver everything" covers:
 ## Flow 2 — Add topics later ("just update the ACLs")
 
 Template: **Add Topics to Application**. Regenerates only
-`gitops/apps/<app>/kong/acl-policy.yaml`. The virtual cluster, credentials and route
+`apps/<app>/kong/acl-policy.yaml` (in the config repo). The virtual cluster, credentials and route
 are untouched, so the app keeps the same bootstrap endpoint and the same credential.
 
 ```mermaid

docs/prerequisites.md

@@ -30,8 +30,9 @@ This repo's `platform/networking/tls-certificate.yaml` then creates a self-signe
 
 ## 2. Argo CD
 
-Drives GitOps: it reconciles `platform/` and auto-onboards each tenant under
-`gitops/apps/*` via the ApplicationSet.
+Drives GitOps: it reconciles this repo's `platform/` and auto-onboards each tenant
+from the companion `kafka-selfservice-gitops` repo's `apps/*` via the ApplicationSet.
+See [`repositories.md`](repositories.md) for the two-repo layout.
 
 ```bash
 kubectl create namespace argocd
@@ -52,16 +53,17 @@ kubectl -n argocd get secret argocd-initial-admin-secret \
   -o jsonpath='{.data.password}' | base64 -d; echo
 ```
 
-Then point Argo CD at your fork and let it manage everything:
+Then apply the Argo manifests **from the `kafka-selfservice-gitops` repo** and let it
+manage everything:
 
 ```bash
-# Edit the repoURL in these files first, then:
-kubectl apply -f gitops/argocd/platform-app.yaml     # reconciles platform/
-kubectl apply -f gitops/argocd/tenants-appset.yaml   # one App per gitops/apps/*
+# In your clone of kafka-selfservice-gitops (edit the repoURLs in these files first):
+kubectl apply -f argocd/platform-app.yaml     # reconciles the portal repo's platform/
+kubectl apply -f argocd/tenants-appset.yaml   # one App per apps/* in the config repo
 ```
 
-If you're not using Argo CD, apply the platform and tenants directly instead:
-`kubectl apply -k platform/` and `kubectl apply -k gitops/apps/<app>/kong/`.
+If you're not using Argo CD, apply directly instead: `kubectl apply -k platform/`
+(portal repo) and `kubectl apply -k apps/<app>/kong/` (config repo).
 
 ## 3. Backstage (in-cluster)
 
@@ -115,8 +117,9 @@ Edit two files in `platform/backstage/`:
 
 - `backstage-secrets.yaml` — set `POSTGRES_PASSWORD`, `GITHUB_TOKEN` (PAT/App token
   with `repo` + PR scope, since the templates open pull requests), and
-  `BACKEND_SECRET` (`openssl rand -base64 32`). Use SealedSecrets / External Secrets
-  for anything real — don't commit plaintext.
+  `BACKEND_SECRET` (`openssl rand -base64 32`). The stub is plaintext for the demo;
+  for anything real, replace it with a Sealed Secret or External Secret —
+  [`docs/secrets.md`](secrets.md) has worked manifests for both.
 - `app-config.configmap.yaml` — point `catalog.locations[].target` at your fork's
   `catalog-info.yaml`. This is what the in-cluster app reads (it replaces
   `backstage/app-config.snippet.yaml`, which remains as a reference for the config

docs/repositories.md

@@ -0,0 +1,72 @@
+# Repository topology
+
+This demo uses **two repositories**. They can be collapsed into one, but the split is
+the more production-realistic setup and is how the templates are wired.
+
+## The two repos
+
+### `kafka-selfservice-backstage` (this repo) — portal / platform
+
+Owned by the platform team; changes rarely.
+
+- `catalog/` — AsyncAPI specs + Backstage catalog entities (the "available topics")
+- `backstage/templates/` — the self-service scaffolder templates
+- `platform/` — the bootstrap: Konnect control plane, Event Gateway data plane,
+  Kafka, networking, and in-cluster Backstage
+- `docs/`, `scripts/`, `examples/`
+
+Backstage's `catalog.locations` points at **this repo's** `catalog-info.yaml` to load
+the templates and the AsyncAPI APIs.
+
+### `kafka-selfservice-gitops` — GitOps config
+
+Written to constantly, by app teams via Backstage PRs.
+
+- `apps/<app>/` — one directory per onboarded application (virtual cluster, ACLs,
+  route, credentials)
+- `argocd/` — the platform `Application` and the tenants `ApplicationSet`
+
+Argo CD watches **this repo's** `apps/*`.
+
+## Who points where
+
+```mermaid
+flowchart LR
+  subgraph portal[kafka-selfservice-backstage]
+    cat[catalog + templates]
+    plat[platform/]
+  end
+  subgraph cfg[kafka-selfservice-gitops]
+    apps[apps/*]
+    appset[argocd/ApplicationSet]
+  end
+  bs[Backstage in-cluster] -->|catalog.locations| cat
+  bs -->|scaffolder PR| apps
+  argo[Argo CD] -->|platform-app -> path platform| plat
+  argo -->|tenants-appset -> apps/*| apps
+```
+
+- **Backstage → portal repo**: reads catalog + templates (`catalog.locations` in
+  `platform/backstage/app-config.configmap.yaml`).
+- **Backstage → config repo**: the templates' `RepoUrlPicker` targets it; PRs add
+  `apps/<app>/`. Nothing is committed back to the portal repo at runtime.
+- **Argo CD → portal repo**: `platform-app.yaml` deploys `platform/`.
+- **Argo CD → config repo**: `tenants-appset.yaml` deploys each `apps/<app>/kong/`.
+
+## Wiring checklist
+
+When you fork/clone, update these to your org's URLs:
+
+| File | Repo | Set to |
+|------|------|--------|
+| `platform/backstage/app-config.configmap.yaml` (`catalog.locations[].target`) | portal | portal repo `catalog-info.yaml` |
+| `argocd/platform-app.yaml` (`spec.source.repoURL`) | config | portal repo |
+| `argocd/tenants-appset.yaml` (both `repoURL`s) | config | config repo |
+| Template runs (`RepoUrlPicker` selection) | — | config repo, chosen per run |
+
+## Prefer a single repo?
+
+Collapse them by putting `apps/` and `argocd/` back into this repo, pointing all the
+`repoURL`s and the templates at this one repo. The trade-off: onboarding-PR churn and
+app-team write access land in the same repo as your platform code. Fine for a solo
+demo; less so once multiple teams onboard.