Skip to content

sandboxed-local-development.md

Latest commit

 

History

History
217 lines (162 loc) · 6.99 KB

sandboxed-local-development.md

File metadata and controls

217 lines (162 loc) · 6.99 KB
Raw

Sandboxed Local Development Environment

A fully self-contained local development environment using Kind and in-cluster Forgejo as the single git source for the entire GitOps loop. This eliminates external dependencies (GitHub, local git daemon, self-signed CA) and provides a simple task sandbox:setup experience.

What It Is

The sandboxed-local environment is a new cluster type alongside local and odcn-production. It runs everything inside a single Kind cluster:

Kind Cluster (rig-sandbox)
├── NGINX Ingress (ports 80/443 → host)
├── CloudNativePG Operator + PostgreSQL
├── CSI VolumeSnapshot CRDs
├── Forgejo (git server, rig-admin/admin1234, 4 repos)
├── ArgoCD (pointing to Forgejo repos)
├── Keycloak (admin/admin1234)
├── MinIO (object storage)
└── Operations Manager

Key differences from the local cluster type:

  • No external git: Forgejo runs in-cluster; ArgoCD reads from Forgejo
  • No self-signed CA: Uses a real wildcard TLS cert for *.sandbox.rijksapp.dev
  • No git daemon: Infrastructure changes are synced to Forgejo via task sandbox:sync
  • Simplified secrets: Plaintext admin/admin1234 credentials where possible

Prerequisites

  1. kind - Kubernetes in Docker
  2. kubectl - Kubernetes CLI
  3. kustomize - Kubernetes configuration management
  4. jq - JSON processor
  5. rsync - File synchronization
  6. Docker - Container runtime
  7. dnsmasq - Local DNS resolver (for wildcard *.sandbox.rijksapp.dev resolution)

DNS Setup (one-time, per developer machine)

Install and start dnsmasq:

brew install dnsmasq
brew services start dnsmasq

The task sandbox:setup command will automatically configure dnsmasq to resolve *.sandbox.rijksapp.dev to 127.0.0.1 and create the macOS resolver file. You can also run this manually:

task sandbox:update-dns

TLS Certificate (every ~90 days, by one team member)

Generate a wildcard certificate:

certbot certonly --manual --preferred-challenges dns -d "*.sandbox.rijksapp.dev"

Copy the certificate files to:

security/tls/sandbox-wildcard/fullchain.pem
security/tls/sandbox-wildcard/privkey.pem

How to Use

Initial Setup

task sandbox:setup

This single command performs the full setup:

  1. Verifies required tools are installed
  2. Activates the sandboxed-local configuration
  3. Generates an AGE key for SOPS encryption
  4. Creates the Kind cluster
  5. Installs NGINX Ingress and CNPG operator
  6. Installs CSI VolumeSnapshot CRDs
  7. Configures CoreDNS for *.sandbox.rijksapp.dev
  8. Creates namespace and imports TLS certificate
  9. Generates SOPS-encrypted secrets
  10. Deploys PostgreSQL and Forgejo (pre-ArgoCD)
  11. Initializes Forgejo with admin user and 4 repositories
  12. Syncs infrastructure to Forgejo
  13. Installs ArgoCD operator and bootstraps the system
  14. Updates /etc/hosts (requires sudo)

Syncing Infrastructure Changes

After modifying files in the infrastructure/ directory:

task sandbox:sync

This copies the infrastructure/ subtree to Forgejo's zad-argo-infrastructure repository. ArgoCD will then detect and apply the changes.

Accessing Services

Service URL Credentials
ArgoCD https://argo.sandbox.rijksapp.dev admin / admin1234
Forgejo https://forgejo.sandbox.rijksapp.dev rig-admin / admin1234
Keycloak https://keycloak.sandbox.rijksapp.dev admin / admin1234
MinIO https://minio.sandbox.rijksapp.dev admin / admin1234
Operations Manager https://zad.sandbox.rijksapp.dev -

Destroying the Cluster

task sandbox:destroy

Cluster Selection

The sandboxed-local environment is available as option 3 in cluster selection:

task select-cluster
# Select option 3: sandboxed-local

Forgejo Repositories

Repository Purpose
zad-projects Project YAML definition files
zad-argo-user-applications ArgoCD Application manifests for user apps
zad-argo-infrastructure Infrastructure config (synced from local infrastructure/ dir)
zad-deployments Kubernetes manifests generated by Operations Manager

Configuration

Environment Variables

The environment is configured via .env-taskfile-sandboxed-local:

CLUSTER_TYPE=sandboxed-local
KIND_CLUSTER_NAME=rig-sandbox
RIG_NAMESPACE=rig-system
INFRASTRUCTURE_CLUSTER_FOLDER=sandboxed-local
BOOTSTRAP_CLUSTER_FOLDER=sandboxed-local

Operations Manager

The Operations Manager connects to Forgejo using in-cluster URLs:

  • Projects: http://forgejo.rig-system.svc.cluster.local:3000/rig-admin/zad-projects.git
  • ArgoCD apps: http://forgejo.rig-system.svc.cluster.local:3000/rig-admin/zad-argo-user-applications.git

Passwords use the plain:admin1234 prefix to bypass AGE decryption.

Optional: Upstream IDP Federation

Keycloak can be configured to federate with an upstream identity provider for SSO. See local-cluster-federation.md for details.

Architecture Notes

Infrastructure Sync

The task sandbox:sync command:

  1. Clones zad-argo-infrastructure from Forgejo
  2. Uses rsync to copy infrastructure/ content (stripping the prefix)
  3. Commits and pushes changes

After sync, the ArgoCD Application path bootstrap/clusters/sandboxed-local maps to the repository root, since the infrastructure/ prefix is stripped.

Secret Management

  • Bootstrap secrets (ArgoCD admin, repo credentials): Plaintext manifests in the overlay
  • Infrastructure secrets (DB passwords, etc.): Generated via _generate-secrets-shared task, SOPS-encrypted with the AGE key
  • Operations Manager secrets: Plaintext Secret (admin/admin1234 for all)
  • sops-age-key: Created from security/key.txt, used by ArgoCD CMP plugin and Operations Manager

TLS

All ingress resources reference the sandbox-wildcard-tls Secret, which contains the real wildcard certificate for *.sandbox.rijksapp.dev. This is imported during setup from security/tls/sandbox-wildcard/.

Troubleshooting

Forgejo not starting

Check if PostgreSQL is ready:

kubectl get cluster rig-db -n rig-system
kubectl logs -n rig-system forgejo-0

ArgoCD not syncing from Forgejo

Verify the repo credential:

kubectl get secrets -n rig-system -l argocd.argoproj.io/secret-type=repository

Check ArgoCD can reach Forgejo:

kubectl exec -n rig-system deployment/argocd-server -- curl -s http://forgejo.rig-system.svc.cluster.local:3000/api/healthz

Infrastructure sync fails

If the ingress route is not yet available, sandbox:sync falls back to port-forwarding. Ensure the Forgejo pod is running:

kubectl get pods -n rig-system -l app=forgejo

DNS not resolving in-cluster

Verify CoreDNS configuration:

kubectl get configmap coredns -n kube-system -o yaml

Certificate issues

Ensure the wildcard cert files exist:

ls -la security/tls/sandbox-wildcard/

The certificate must be valid for *.sandbox.rijksapp.dev.