[DX-110047] Helm Chart for Dremio MCP Server Kubernetes Deployment

[aniket-s-kulkarni]

Overview

This PR implements a production-ready Helm chart for deploying the Dremio MCP Server on Kubernetes with support for OAuth authentication, streaming HTTP mode, and enterprise security features.

Jira Ticket: DX-110047
Parent Epic: DX-107224 - Roll MCP server out to DC

What This PR Does

This PR adds comprehensive Kubernetes deployment support for the Dremio MCP Server through a Helm chart that enables:

Key Features

• Streaming HTTP Mode - Web-based deployment support for chat frontends
• Horizontal Pod Autoscaling (HPA) - Automatic scaling based on CPU/memory
• Prometheus Metrics Integration - Built-in observability
• Ingress Support - TLS/SSL termination and routing
• Security Best Practices - Non-root user, read-only filesystem, dropped capabilities
• Flexible Authentication - OAuth (recommended), PAT, or existing secrets

How It Works

Architecture

The Helm chart deploys the MCP server with a file-based configuration approach:

1. ConfigMap - Stores config.yaml with:

   • Dremio URI
   • Tools configuration (server mode)
   • Metrics settings
   • DML permissions

2. Secret (optional) - Stores PAT when provided:

   • Created automatically from inline PAT value
   • Or references existing Kubernetes secret
   • Mounted as read-only file at /app/secrets/pat

3. Deployment - Runs MCP server with:

   • Command: dremio-mcp-server run --cfg /app/config/config.yaml --enable-streaming-http --no-log-to-file
   • Read-only volume mounts for config and secrets
   • Security context (non-root, read-only filesystem)
   • Health checks and resource limits

Authentication Flow (OAuth - Recommended)

sequenceDiagram
    participant User
    participant ChatFrontend as Chat Frontend
    participant IdP as OAuth IdP
    participant Dremio as Dremio API
    participant MCP as MCP Server

    User->>ChatFrontend: 1. Initiate chat session
    ChatFrontend->>IdP: 2. Redirect to OAuth login
    IdP->>User: 3. Present login page
    User->>IdP: 4. Authenticate
    IdP->>ChatFrontend: 5. Return OAuth token
    ChatFrontend->>Dremio: 6. Exchange token (External Token Provider)
    Dremio->>ChatFrontend: 7. Return Dremio access token
    ChatFrontend->>MCP: 8. Send request with Dremio token
    MCP->>Dremio: 9. Execute operations with user context
    Dremio->>MCP: 10. Return results
    MCP->>ChatFrontend: 11. Return response
    ChatFrontend->>User: 12. Display results

Loading

Configuration Modes

The chart supports three authentication modes:

1. OAuth + External Token Provider (Production - Recommended)

helm install dremio-mcp ./helm/dremio-mcp \
  --set dremio.uri=https://dremio.example.com:9047

• No PAT required in the chart
• Chat frontend handles OAuth flow and token exchange
• Passes Dremio token to MCP server via Authorization header
• See AUTHENTICATION.md for complete implementation guide

2. Personal Access Token (Development/Testing)

helm install dremio-mcp ./helm/dremio-mcp \
  --set dremio.uri=https://dremio.example.com:9047 \
  --set dremio.pat=<your-pat>

• PAT stored in auto-generated Kubernetes secret
• Mounted as file and referenced in config.yaml
• Not recommended for production

3. Existing Secret

helm install dremio-mcp ./helm/dremio-mcp \
  --set dremio.uri=https://dremio.example.com:9047 \
  --set dremio.existingSecret=my-secret

• References pre-existing Kubernetes secret
• Secret must have a pat key
• Useful for GitOps workflows

Files Added/Modified

Helm Chart Structure

helm/dremio-mcp/
├── Chart.yaml                    # Chart metadata
├── values.yaml                   # Default configuration values
├── templates/
│   ├── _helpers.tpl             # Template helpers
│   ├── configmap.yaml           # Dremio configuration
│   ├── secret.yaml              # PAT secret (conditional)
│   ├── deployment.yaml          # MCP server deployment
│   ├── service.yaml             # Kubernetes service
│   ├── serviceaccount.yaml      # Service account
│   ├── ingress.yaml             # Ingress (optional)
│   ├── hpa.yaml                 # Horizontal Pod Autoscaler (optional)
│   └── NOTES.txt                # Post-install instructions
├── examples/
│   ├── values-oauth-production.yaml      # Production OAuth setup
│   ├── values-onprem.yaml                # On-premises deployment
│   ├── values-with-pat.yaml              # Development with PAT
│   └── values-with-existing-secret.yaml  # Using existing secrets
├── README.md                    # Complete installation guide
├── AUTHENTICATION.md            # OAuth implementation guide with code
├── QUICKSTART.md               # Quick start guide
└── test-chart.sh               # Shell-based validation tests

Tests

• tests/test_chart.py - Pytest-based Helm chart tests:
  • Helm lint validation
  • OAuth mode configuration
  • Inline PAT mode
  • Existing secret mode
  • Metrics and DML configuration
  • Security settings validation
  • Example values files rendering
  • 12 test classes with comprehensive coverage

Security Features

The Helm chart implements enterprise security best practices:

Pod Security

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1001
  fsGroup: 1001

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: true

Configuration Reference

Key Values

Parameter	Description	Default
dremio.uri	Dremio Software instance URI	"" (required)
dremio.pat	Personal Access Token (dev/test only)	""
dremio.existingSecret	Name of existing secret with PAT	""
dremio.allowDml	Allow DML operations (create views)	false
mcp.enableStreamingHttp	Enable streaming HTTP mode	true
mcp.port	MCP server port	8000
tools.serverMode	Tool types to enable	FOR_DATA_PATTERNS
metrics.enabled	Enable Prometheus metrics	false
metrics.port	Metrics port	9091
autoscaling.enabled	Enable HPA	false
ingress.enabled	Enable ingress	false

See helm/dremio-mcp/values.yaml for complete reference.

Monitoring and Observability

Prometheus Metrics

metrics:
  enabled: true
  port: 9091

Exposes metrics at /metrics endpoint for Prometheus scraping.

Health Checks

• Liveness Probe: GET /healthz every 30s
• Initial Delay: 10s
• Timeout: 5s
• Failure Threshold: 3

Logging

• Logs to stdout/stderr (Kubernetes best practice)
• JSON logging support via mcp.logging.enableJson
• Log level configurable via mcp.logging.level

[alex-aidun-dremio]

Left one comment to remove a Cloud reference since we are already hosting the MCP server in Cloud