Infrastructure: Define a Kubernetes Cluster with Pulumi

[edwardchalstrey1]

Development started on the kubernetes branch.

• Pre-requisite learning:
  • Kubernetes with kind or minikube
  • Pulumi - Kubernetes tutorial
• Local Infrastructure (Implemented in infra/)
  • Define a local Kubernetes cluster layout with Pulumi to deploy the Dagster, Streamlit, and Postgres services.
• Cloud Provider Deployment Readiness
  • Define Cloud Cluster Environment: Extend the existing Pulumi code to provision a managed Kubernetes cluster on a cloud provider (e.g., Google Kubernetes Engine (GKE) or Amazon EKS).
  • Expose UI Services externally: Update the network configuration in infra/__main__.py to change the Service types for Dagster and Streamlit from ClusterIP to LoadBalancer (or setup an Ingress controller) to assign public IPs.
  • Private Image Repositories: Configure kubernetes.io/dockerconfigjson secrets in Pulumi to ensure the cloud cluster is authorized to pull the custom images from Docker Hub or a private container registry.
• Production Data & State Persistence Architecture
  • Managed Postgres Migration: In a production cloud setting, Postgres shouldn't run inside Kubernetes. Provision a managed database (like AWS RDS or Google Cloud SQL with pgvector enabled) using Pulumi, and update the application pods to connect to this external database instance.
  • Dagster Data Handling & Storage:
    • Currently, raw data and parquet files are downloaded and saved to disk within the container. On the cloud, ephemeral pod storage will be lost on container restart.
    • Solution: Reconfigure Dagster's I/O managers to store intermediate parquet assets in a cloud object store (e.g., AWS S3 or GCS) instead of the local filesystem data/ directory.
  • Dagster Instance State Management: Configure the Dagster Helm chart (or equivalent Pulumi components) to back its run history/schedule databases with the persistent Managed Postgres instance instead of the default ephemeral SQLite state.