Epic:  Full Stack CICD build and deployment of MCP CF through single configuration

# MCP Stack Deployment System - Proposal Summary

## 🎯 Overview

This feature provides a **unified deployment system** for MCP Context Forge that builds and deploys all containers (gateway + plugins) from a **single configuration file** (`mcp-stack.yaml`).

### What It Does

- **Builds** all plugin containers from git repositories
- **Generates** mTLS certificates automatically
- **Creates** deployment manifests (Kubernetes or Docker Compose)
- **Deploys** the complete stack (gateway + all plugins)
- **Integrates** with CI/CD pipelines using a builder container
- **Supports** hybrid execution: Dagger (optimized) or plain Python (portable)

### One Command Deployment

```bash
./scripts/mcp-deploy.py deploy mcp-stack.yaml
```

This single command orchestrates the entire deployment lifecycle from source code to running services.

---

## 📋 Example Configuration File

Here's a complete `mcp-stack.yaml` that defines an entire MCP deployment:

```yaml
# Deployment target configuration
deployment:
  type: kubernetes          # or 'compose' for Docker Compose
  namespace: mcp-gateway    # Kubernetes namespace

# MCP Gateway configuration
gateway:
  # Container image (from registry or local build)
  image: ghcr.io/ibm/mcp-context-forge:latest
  image_pull_policy: IfNotPresent

  # Networking
  port: 4444
  service_type: ClusterIP
  service_port: 4444

  # Resource limits (Kubernetes)
  replicas: 2
  memory_request: 512Mi
  memory_limit: 1Gi
  cpu_request: 250m
  cpu_limit: 1000m

  # Environment configuration
  env_file: gateway.env
  env_vars:
    LOG_LEVEL: INFO
    MCPGATEWAY_UI_ENABLED: "true"
    MCPGATEWAY_ADMIN_API_ENABLED: "true"
    MCPGATEWAY_A2A_ENABLED: "true"
    DATABASE_URL: postgresql://user:pass@postgres:5432/mcpgateway
    REDIS_URL: redis://redis:6379
    # Secrets injected from CI/CD vault:
    JWT_SECRET_KEY: ${JWT_SECRET_KEY}

  # mTLS security
  mtls_enabled: true

# External plugins - built from git repositories
plugins:
  # OPA Policy Filter Plugin
  - name: OPAPluginFilter

    # Git repository to build from
    repo: https://github.com/yourorg/mcp-opa-plugin.git
    ref: main
    containerfile: Containerfile

    # Or use pre-built image:
    # image: ghcr.io/yourorg/mcp-opa-plugin:latest

    # Networking
    port: 8000
    service_type: ClusterIP

    # Resources
    replicas: 2
    memory_request: 256Mi
    memory_limit: 512Mi
    cpu_request: 100m
    cpu_limit: 500m

    # Environment
    env_file: plugins/opa.env
    env_vars:
      LOG_LEVEL: DEBUG
      OPA_POLICY_PATH: /app/policies

    # Security
    mtls_enabled: true
    priority: 10

  # ClamAV Virus Scanning Plugin
  - name: ClamAVPlugin
    repo: https://github.com/yourorg/mcp-clamav-plugin.git
    ref: v1.0.0

    port: 3000
    replicas: 1
    memory_request: 512Mi
    memory_limit: 1Gi

    env_file: plugins/clamav.env
    env_vars:
      CLAMAV_HOST: clamav-server
      CLAMAV_PORT: "3310"

    mtls_enabled: true

# Certificate configuration
certificates:
  validity_days: 825      # ~2.25 years
  auto_generate: true
  ca_path: ./certs/mcp/ca
  gateway_path: ./certs/mcp/gateway
  plugins_path: ./certs/mcp/plugins

# CI/CD integration
ci_cd:
  # Secrets to inject from vault (not stored in repo)
  vault_secrets:
    - JWT_SECRET_KEY
    - DATABASE_PASSWORD
    - REDIS_PASSWORD
    - API_KEY

  # Build options
  build:
    parallel: true        # Parallel plugin builds
    cache: true           # Enable build caching

  # Deployment options
  deploy:
    wait: true           # Wait for deployment ready
    timeout: 300         # Timeout in seconds
    verify: true         # Run health checks
```

---

## 📁 Repository Structure

Here's what a deployment repository looks like using this system:

```
my-mcp-deployment/
├── mcp-stack.yaml              # Main configuration (single source of truth)
│
├── gateway.env                 # Gateway environment variables
│   └── (references ${SECRETS} for vault injection)
│
├── plugins/
│   ├── opa.env                # OPA plugin environment
│   └── clamav.env             # ClamAV plugin environment
│
├── .github/workflows/          # CI/CD pipeline
│   └── deploy.yml             # GitHub Actions workflow (10-20 lines)
│
├── certs/                     # Auto-generated (not in git)
│   └── mcp/
│       ├── ca/                # Certificate Authority
│       ├── gateway/           # Gateway client cert
│       └── plugins/           # Per-plugin server certs
│           ├── OPAPluginFilter/
│           └── ClamAVPlugin/
│
├── deploy/                    # Auto-generated (not in git)
│   └── manifests/
│       ├── plugins-config.yaml           # Auto-generated plugin config
│       ├── gateway-deployment.yaml       # Kubernetes manifests
│       ├── plugin-opa-deployment.yaml
│       └── plugin-clamav-deployment.yaml
│
└── README.md                  # Deployment instructions
```

**Files in Git:**
- ✅ `mcp-stack.yaml` - Configuration
- ✅ `*.env` - Environment templates (with `${VARIABLE}` placeholders)
- ✅ `.github/workflows/` - CI/CD pipeline
- ✅ `.gitignore` - **MUST exclude `certs/` directory!**
- ❌ `certs/` - **NEVER commit! Generated at deploy time or from vault**
- ❌ `deploy/` - Auto-generated at deploy time
- ❌ Secrets - Injected from vault

**⚠️ Security Note:** The `certs/` directory contains sensitive cryptographic material:
- `certs/mcp/ca/ca.key` - **CA private key (CRITICAL - NEVER commit!)**
- `certs/mcp/ca/ca.crt` - CA certificate (public, but still shouldn't be in git)
- Gateway and plugin certificates

**Best Practices:**
1. **Development/Testing**: Generate certificates locally, add `certs/` to `.gitignore`
2. **Production**: Store CA key in **vault** (HashiCorp Vault, AWS Secrets Manager, etc.)
3. **CI/CD**: Generate fresh certificates each deployment OR retrieve CA from vault and generate signed certs

---

## 🔍 Configuration Breakdown

### Deployment Section
```yaml
deployment:
  type: kubernetes          # Target: 'kubernetes' or 'compose'
  namespace: mcp-gateway    # K8s namespace (or compose project name)
```
Defines **where** to deploy (local Docker Compose or production Kubernetes).

### Gateway Section
```yaml
gateway:
  image: ghcr.io/ibm/mcp-context-forge:latest
  port: 4444
  replicas: 2
  env_vars:
    DATABASE_URL: postgresql://...
    JWT_SECRET_KEY: ${JWT_SECRET_KEY}  # From vault
  mtls_enabled: true
```
Defines the **MCP Gateway** configuration with resources, networking, and security.

### Plugins Section
```yaml
plugins:
  - name: OPAPluginFilter
    repo: https://github.com/yourorg/mcp-opa-plugin.git  # Build from source
    ref: main
    port: 8000
    mtls_enabled: true
```
Defines **external plugins** to build and deploy. Each plugin:
- Can be built from a git repository OR use a pre-built image
- Gets its own container, service, and mTLS certificates
- Auto-registers with the gateway via generated `plugins-config.yaml`

### Auto-Generated Files

The system automatically generates:

**1. `plugins-config.yaml`** - Gateway plugin configuration
```yaml
plugins:
- name: OPAPluginFilter
  kind: external
  mcp:
    proto: STREAMABLEHTTP
    url: https://mcp-plugin-opapluginfilter.mcp-gateway.svc:8000/mcp
- name: ClamAVPlugin
  kind: external
  mcp:
    proto: STREAMABLEHTTP
    url: https://mcp-plugin-clamavplugin.mcp-gateway.svc:3000/mcp
```
URLs are automatically computed based on:
- **Deployment type**: Docker hostnames for Compose, K8s service DNS for Kubernetes
- **mTLS setting**: HTTPS when enabled, HTTP otherwise
- **Port configuration**: From plugin definition

**2. Deployment Manifests** - Kubernetes or Docker Compose
- **Kubernetes**: Deployments, Services, Secrets, ConfigMaps
- **Docker Compose**: Services, networks, volumes, health checks

**3. mTLS Certificates** - Complete PKI infrastructure
- CA certificate (shared trust anchor)
- Gateway client certificate (connects to plugins)
- Per-plugin server certificates (unique for each plugin)

---

## 🏗️ CI/CD Integration

### GitHub Actions Example

```yaml
name: Deploy MCP Stack

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to Production
        run: |
          docker run --rm \
            -v $PWD:/workspace \
            -v ~/.kube:/root/.kube:ro \
            -w /workspace \
            ghcr.io/ibm/mcp-builder:latest \
            deploy mcp-stack.yaml
        env:
          JWT_SECRET_KEY: ${{ secrets.JWT_SECRET_KEY }}
          DATABASE_PASSWORD: ${{ secrets.DATABASE_PASSWORD }}
          API_KEY: ${{ secrets.API_KEY }}
```

**10 lines of pipeline code** vs typical 100+ lines for manual deployment.

### Builder Container

The `mcp-builder` container includes all deployment tools:
- Python 3.12
- Dagger CLI (for optimized builds)
- Docker CLI
- kubectl (Kubernetes)
- helm
- docker-compose
- OpenSSL (certificate generation)
- Git

**Two execution modes:**
1. **Dagger Mode** (when Dagger available): Parallel builds, automatic caching, content-addressable storage
2. **Plain Python Mode** (fallback): Sequential builds using docker/podman, works everywhere

The system auto-detects and uses the best available option.

---

## ✨ Key Benefits

### 1. Single Source of Truth
- **One file** defines the entire deployment
- No configuration duplication or drift
- Easy to version control and review changes
- Clear understanding of the complete stack

### 2. Automated Configuration Generation
- **Plugin URLs** computed automatically based on deployment type
- **mTLS certificates** generated for all components
- **Deployment manifests** created from templates
- Eliminates manual configuration errors

### 3. Multi-Environment Support
- **Same configuration file** works for:
  - Local development (Docker Compose)
  - Staging (Kubernetes)
  - Production (Kubernetes)
- Just change `deployment.type` field
- Environment-specific values via `.env` files

### 4. Git-Based Plugin Deployment
- Plugins built **directly from git repositories**
- Specify branch, tag, or commit (`ref: main`, `ref: v1.0.0`)
- Or use pre-built images for faster deployments
- Full traceability from source to deployment

### 5. Integrated Security (mTLS)
- **Automatic certificate generation** for all components
- Per-plugin isolation (unique server certificates)
- Certificate rotation support built-in
- Deployed as Kubernetes Secrets or Docker volumes
- Zero-trust architecture

**Certificate Management Options:**

**Option A: Ephemeral Certificates (Development)**
```bash
# Generate fresh certs each deployment (local dev)
mcp-deploy certs mcp-stack.yaml
mcp-deploy deploy mcp-stack.yaml
# Certificates discarded after deployment
```

**Option B: Vault-Managed CA (Production)**
```bash
# Retrieve CA from vault, generate signed certs
export CA_CERT=$(vault kv get -field=cert secret/mcp/ca)
export CA_KEY=$(vault kv get -field=key secret/mcp/ca)
mcp-deploy certs mcp-stack.yaml --ca-from-env
mcp-deploy deploy mcp-stack.yaml
```

**Option C: External PKI Integration**
```bash
# Use existing PKI (cert-manager, Venafi, etc.)
mcp-deploy deploy mcp-stack.yaml --external-pki
# System generates CSRs, submits to PKI for signing
```

**Security Best Practices:**
- ✅ CA private key stored in vault (HashiCorp Vault, AWS Secrets Manager, K8s Secret)
- ✅ Certificates generated per-deployment with short validity (90 days)
- ✅ Automatic rotation before expiry
- ✅ `.gitignore` includes `certs/` directory
- ❌ **NEVER** commit `certs/` to git
- ❌ **NEVER** share CA private keys

### 6. CI/CD Native
- **Builder container** provides consistent environment
- **Vault integration** for secrets (GitHub Secrets, GitLab Variables, HashiCorp Vault)
- Secrets never committed to repository
- Simple pipeline definitions (10-20 lines)
- Works with: GitHub Actions, GitLab CI, Jenkins, etc.

### 7. Hybrid Execution Model
- **Dagger mode**: Fast parallel builds with automatic caching
- **Plain Python mode**: Works everywhere, zero external dependencies
- Auto-detection with manual override (`--no-dagger`)
- Same results regardless of execution mode

### 8. Developer Experience
- **Quick start**: One command to deploy entire stack
- **Fast iteration**: Change config, redeploy
- **Clear errors**: Validation before deployment
- **Verbose mode**: Debugging support
- **Dry-run**: Preview changes without deploying

### 9. Scalability
- Deploy **any number of plugins**
- Parallel plugin builds (with Dagger)
- Resource limits per component
- Horizontal scaling (replicas)
- Production-ready defaults

### 10. Flexibility
- **Kubernetes** for production
- **Docker Compose** for local development
- Mix pre-built images and git-based builds
- Custom environment variables per plugin
- Extensible template system

---

## 🚀 Usage Examples

### Local Development
```bash
# Deploy locally with Docker Compose
python3 scripts/mcp-deploy.py deploy mcp-stack.yaml
```

### Production Kubernetes
```bash
# Generate and review manifests first
python3 scripts/mcp-deploy.py generate mcp-stack.yaml -o ./deploy/staging
cat ./deploy/staging/plugins-config.yaml

# Deploy to cluster
python3 scripts/mcp-deploy.py deploy mcp-stack.yaml

# Verify deployment
python3 scripts/mcp-deploy.py verify mcp-stack.yaml --wait --timeout 300
```

### CI/CD Pipeline
```bash
# Using builder container (includes all tools)
docker run --rm \
  -v $PWD:/workspace \
  -v ~/.kube:/root/.kube:ro \
  -w /workspace \
  ghcr.io/ibm/mcp-builder:latest \
  deploy mcp-stack.yaml
```

### Build Plugins Only
```bash
# Build all plugins from git repositories
python3 scripts/mcp-deploy.py build mcp-stack.yaml --plugins-only

# Build specific plugin
python3 scripts/mcp-deploy.py build mcp-stack.yaml --plugin OPAPluginFilter
```

### Certificate Management
```bash
# Generate mTLS certificates for all plugins
python3 scripts/mcp-deploy.py certs mcp-stack.yaml

# Check certificate expiry
make certs-mcp-check
```

---

## 🎓 How It Works

1. **Parse Configuration**: Load and validate `mcp-stack.yaml`
2. **Build Plugins**: Clone git repos and build containers (parallel with Dagger)
3. **Generate Certificates**: Create CA + gateway client cert + plugin server certs
4. **Generate Manifests**:
   - `plugins-config.yaml` with environment-specific URLs
   - Kubernetes manifests OR Docker Compose file
5. **Deploy**: Apply manifests to target environment
6. **Verify**: Health checks and readiness validation

All from a single configuration file and single command.

---

## 🤔 Questions for Community

1. **Configuration Schema**: Is this the right level of abstraction?
2. **Plugin Sources**: Should git-based builds be the default or pre-built images?
3. **Deployment Targets**: Need support for other platforms (Terraform, CloudFormation)?
4. **Certificate Management**: Should auto-rotation be included in v1?
5. **Multi-Cluster**: Support for deploying to multiple clusters from one config?

---

**Ready for feedback and discussion!** 🚀

Epic: Full Stack CICD build and deployment of MCP CF through single configuration #1148

Description

MCP Stack Deployment System - Proposal Summary

🎯 Overview

What It Does

One Command Deployment

📋 Example Configuration File

📁 Repository Structure

🔍 Configuration Breakdown

Deployment Section

Gateway Section

Plugins Section

Auto-Generated Files

🏗️ CI/CD Integration

GitHub Actions Example

Builder Container

✨ Key Benefits

1. Single Source of Truth

2. Automated Configuration Generation

3. Multi-Environment Support

4. Git-Based Plugin Deployment

5. Integrated Security (mTLS)

6. CI/CD Native

7. Hybrid Execution Model

8. Developer Experience

9. Scalability

10. Flexibility

🚀 Usage Examples

Local Development

Production Kubernetes

CI/CD Pipeline

Build Plugins Only

Certificate Management

🎓 How It Works

🤔 Questions for Community

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions