Configuration Export Feature for GitOps Support

[rajithacharith]

Overview

This proposes adding configuration export capabilities to Thunder to support a GitOps workflow where configurations created in one environment can be exported and deployed to other environments.

Background

Thunder will support two runtime modes:

1. Database Mode: Configurations managed via UI/API, stored in a database
2. File Mode: Configurations loaded from YAML files at runtime

Typical Workflow

Developer Environment
    ↓
  Create resources via UI/API
    ↓
  Export configurations as YAML
    ↓
  Commit to Git repository
    ↓
Production Environment 
    ↓
  Deploy with YAML configs mounted
  Environment variables injected for secrets/env-specific values

Goals

1. Enable exporting configurations from database mode as YAML files
2. Support parameterization for secrets and environment-specific values
3. Allow the same configuration to run in different environments with different variable values
4. Support both runtime modes in the same codebase

This proposal focuses on providing export functionality.

---

Key Design Decisions

1. Parameterization Strategy

Secrets and environment-specific values will be parameterized and resolved from environment variables at runtime.

# Exported YAML (committed to Git)
application:
  name: my-app
  clientSecret: ${{APP_MY_APP_CLIENT_SECRET}}
  redirectUri: ${{REDIRECT_URI}}

# Dev environment variables
APP_MY_APP_CLIENT_SECRET=dev-secret-123
REDIRECT_URI=https://dev.example.com/callback

# Prod environment variables
APP_MY_APP_CLIENT_SECRET=prod-secret-xyz
REDIRECT_URI=https://prod.example.com/callback

---

2. Auto-Parameterization Rules

Which fields should be automatically parameterized during export?

• Parameterized variables are defined per resource.
• Support optional rules or a param-rules.yaml for custom field mapping.
• Possible CLI flag:

  generator export --parameterize=auto|manual|none

---

3. Export Scope & Filtering

Filtering capabilities should allow exporting subsets of resources.
IF CLI

# Proposed CLI commands
generator export --all                                            # Everything
generator export --type applications                              # By resource type
generator export --type applications --names app1,app2            # Specific resources

If Export using API

curl -X POST "https://localhost:8090/api/v1/export" \
  -H "Content-Type: application/json" \
  -H "Accept: application/zip" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "resources": {
      "applications": ["app1", "app2"],
      "identity-providers": ["google-oauth"]
    },
  }' \
  -o "thunder-export-selective.zip"

---

4. Output Structure

Option 1: Directory Structure (Preferred)

immutable_resources/
├── applications/
│   ├── app1.yaml
│   └── app2.yaml
├── identity-providers/
│   ├── google-oauth.yaml
│   └── okta-saml.yaml

Option 2: Single File per Export

thunder-export-2025-10-24.yaml

Suggested Approach:
As the runtime expects organized directories, the download ZIP should follow this structure for direct use with Thunder runtime.

---

5. Parameter Resolution

When are parameters resolved?

# Option A: Resolved at application startup
clientSecret: ${CLIENT_SECRET}  # Resolved once at boot

# Startup will fail if environment variable is missing

---

6. Parameter Documentation

Automatically generate parameter documentation for transparency and CI/CD validation.

# parameters.yaml
parameters:
  - name: APP_MY_APP_CLIENT_SECRET
    description: "OAuth client secret for my-app"
    required: true
    type: secret
    usedIn:
      - applications/my-app.yaml
      
  - name: REDIRECT_URI
    description: "OAuth redirect URI"
    required: true
    type: string
    defaultValue: "http://localhost:8080/callback"
    usedIn:
      - applications/my-app.yaml

# .env.example
# Generated by Thunder Export on 2025-10-24

# Application: my-app
APP_MY_APP_CLIENT_SECRET=
REDIRECT_URI=https://example.com/callback

# Identity Provider: google-oauth
IDP_GOOGLE_OAUTH_CLIENT_SECRET=
IDP_GOOGLE_OAUTH_CLIENT_ID=

---

⚖️ Proposed Approaches

Approach Comparison Summary

Approach	Description	Pros	Cons	Best For
CLI Tool	Command-line utility that exports configurations using Thunder APIs	- CI/CD & GitOps friendly - No new API surface - Fine-grained filtering	- Requires installation/versioning - Slightly higher learning curve	Automated pipelines
Download Endpoint	REST API endpoint providing a downloadable ZIP of YAMLs	- No extra tool install - UI integration possible - Easy for one-off exports	- Less automation-friendly - File size limits	Ad-hoc/manual exports
Hybrid	CLI consumes the export endpoint as its backend	- Combines benefits of both - Enables UI integration - Unified implementation	- Slightly higher complexity	Teams needing both UI and CI/CD support

Visual Overview

+-------------+          +-------------------+
|   Thunder   | <------> |  Export Endpoint  |
+-------------+          +-------------------+
         ↑                        ↑
         |                        |
   [ CLI Tool ]             [ Web UI Export ]
         ↓
     YAML Files → Git → File Mode Runtime

Example Usage

Developer Workflow (DB Mode → File Mode)

# 1. Developer creates resources in UI (DB mode)
# 2. Export configurations
generator export --output ./configs --all

# 3. Review generated files
ls -la configs/
# applications/  identity-providers/  .env.example  parameters.yaml

# 4. Commit to Git
git add configs/
git commit -m "Add Thunder configurations"
git push

# 5. Deploy to production (File mode)
kubectl create configmap thunder-config --from-file=./configs
kubectl create secret generic thunder-env --from-env-file=.env.prod
kubectl apply -f thunder-deployment.yaml

Environment-Specific Deployment

# Development
export APP_MY_APP_CLIENT_SECRET=dev-secret
export REDIRECT_URI=https://dev.example.com/callback
./thunder --mode=file --config=./configs

# Production
export APP_MY_APP_CLIENT_SECRET=prod-secret
export REDIRECT_URI=https://prod.example.com/callback
./thunder --mode=file --config=./configs

API-Based Export Workflow

1. Export All Resources via API

# Export all configurations
curl -X GET "https://localhost:8090/api/v1/export" \
  -H "Accept: application/zip" \
  -H "Authorization: Bearer <token>" \
  -o "thunder-export-all.zip"

# Extract and review
unzip thunder-export-all.zip -d ./configs
ls -la configs/
# applications/  identity-providers/  .env.example  parameters.yaml

2. Export Specific Resource Types

# Export only applications
curl -X GET "https://localhost:8090/api/v1/export?type=applications" \
  -H "Accept: application/zip" \
  -H "Authorization: Bearer <token>" \
  -o "thunder-export-applications.zip"

# Export multiple resource types
curl -X GET "https://localhost:8090/api/v1/export?type=applications,identity-providers" \
  -H "Accept: application/zip" \
  -H "Authorization: Bearer <token>" \
  -o "thunder-export-multi-types.zip"

3. Export with Advanced Filtering (POST API)

# Export specific named resources with custom parameterization
curl -X POST "https://localhost:8090/api/v1/export" \
  -H "Content-Type: application/json" \
  -H "Accept: application/zip" \
  -H "Authorization: Bearer <token>" \
  -d '{
    "resources": {
      "applications": ["my-app", "admin-portal"],
      "identity-providers": ["google-oauth", "azure-ad"]
    },
    "parameterize": "auto",
    "includeDocumentation": true,
    "outputFormat": "zip"
  }' \
  -o "thunder-export-selective.zip"

[senthalan]

This GitOps export capability will significantly improve our production deployment workflow. I support the overall direction, but have some critical questions about implementation details that should be clarified before we proceed.

1. Can we include versions?

To handle schema evolution gracefully, we should version all exported configurations. This makes future breaking changes manageable and enables validation easier.

apiVersion: thunder.io/v1
kind: ApplicationConfiguration
metadata:
  exportedAt: "2025-10-24T10:30:00Z"
  thunderVersion: "2.1.0"
  schemaVersion: "1.0"

I have propose to use Kubernetes Custom Resource Definition (CRD) format if possible.

2. Parameterization Rules should be controlled by Thunder, not users

The proposal suggests customizable param-rules.yaml, but I question whether
this flexibility is necessary.

Concerns:

• Increases complexity for users
• Could lead to inconsistent configurations across teams
• Makes upgrades harder if each team has custom rules

Recommendation: Thunder should define standard parameterization per resource type. If customization is truly needed, can you provide specific use cases that justify this complexity?

3. Use POST for all export operations

The proposal uses both GET and POST endpoints. We should standardize on POST for all exports because:

• GET URLs have length limits that break with many resource names
• POST body allows future extensibility without API changes
• Consistent pattern is easier for API consumers to learn

Unless there's a strong reason to support GET, let's simplify to POST-only.

4. Async export is critical for large environments

For production environments with hundreds of resources, synchronous export will likely timeout. We need:

• Async job creation: POST returns job ID immediately
• Status endpoint: GET /api/v1/export/jobs/{jobId}/status
• Download when ready: GET /api/v1/export/jobs/{jobId}/download

Question: What's the expected maximum dataset size we need to support?
This will help define whether async is required or optional.

5. Parameter naming strategy to prevent conflicts

The proposal shows examples like:

clientSecret: ${{APP_MY_APP_CLIENT_SECRET}}
redirectUri: ${{REDIRECT_URI}}

Concerns:

• REDIRECT_URI is too generic - multiple apps could need different values
• No clear namespacing strategy for parameters across resources
• Risk of accidental overwrites when merging configs from different teams

Recommendation: Enforce consistent parameter naming convention:

[rajithacharith]

Currently, when configurations are exported, the UUID associated with each resource is not exported.
As a result, when the configuration is re-imported or deployed in a different environment (e.g., another pod), the backend automatically generates a new UUID.

This leads to inconsistency issues — the same logical application or resource ends up with different UUIDs across deployments or pods.
Such duplication causes problems in scenarios like synchronization, cross-references, or when UUIDs are used as stable identifiers in integrations or downstream systems.

Proposed Approaches

1. Export and reuse existing UUIDs

Description:
Include the existing UUIDs in the export so that when configurations are imported elsewhere, the same UUIDs are preserved instead of being regenerated.

Pros:

• Keeps identifiers consistent across environments.
• Minimal code change needed on the import/export side.
• Simplifies tracking and correlation across multiple systems.

Cons:

• Potential risk of UUID collision if the same resource type exists in different systems/environments.
• Makes it harder to create independent copies of resources — duplicates will share the same UUID.
• Requires careful handling to avoid accidentally overwriting existing resources with the same UUID.

And also, this will end up having the same uuid for application in higher environments, that can lead to an issue as this will let the developers (users with lower level access) knows the uuids of the applications in higher environments.

2. Parameterize the UUID

Description:
Make UUIDs configurable parameters in the configuration. During deployment or import, the UUID can be overridden through parameter substitution.

Pros:

• Allows controlled reuse of UUIDs where needed.
• Avoids hardcoding of UUIDs in exported files.
• Works well with automated pipelines or templating systems (e.g., Helm, Terraform).

Cons:

• Adds complexity — UUIDs must be explicitly managed as parameters.
• Risk of configuration errors if parameters are not set consistently.
• Slightly more effort for users to maintain parameter files.

References needs to be properly managed as if one exported file needs the UUID of the other files.

Sometimes users needs to edit the exported files to match the UUID to the existing available UUID to match the dependencies.

3. Introduce a new user-defined attribute: handle

Description:
Add a new human-readable, user-defined identifier (e.g., handle) to uniquely identify a resource across environments.
The backend can still generate a UUID internally, but all cross-references and logical mappings will use the handle.

Pros:

• Stable across environments — user controls the handle name.
• Avoids UUID duplication issues entirely.
• Easier to recognize and reference (e.g., app: "orders-service" instead of UUID).
• Cleaner separation between internal (system) IDs and external (logical) IDs.

Cons:

• Requires schema changes to introduce a new attribute.
• Adds an extra field to manage (UUID + handle coexistence).
• Need to enforce uniqueness and validation of handles across resources.

Currently we do not support handle pattern for any of the resources.

[rajithacharith]

When exporting and importing applications across different environments (e.g., development, staging, production), redirect URIs need to be environment-specific. However, the current implementation faces a fundamental limitation:
Redirect URIs are stored as arrays in the application configuration
Environment variables cannot directly represent arrays, making it impossible to parameterize multiple redirect URIs in a straightforward manner
This creates a blocker for environment-agnostic application configurations and automated deployment pipelines.

Proposed solutions

Solution 1: Comma-Separated String with Split Logic
Approach: Store redirect URIs as a comma-separated string in environment variables and split them into an array when building the application object.
Implementation:

• Create new struct definitions specifically for reading from YAML configuration files (instead of reusing the application request structs)
• Parse the comma-separated environment variable value during application build/initialization
• Apply this parsing logic exclusively to redirect URIs

Cons:

• Requires new struct definitions, increasing codebase complexity
• Only solves the problem for redirect URIs; other array fields would need separate handling
• Comma-separated format may cause issues if URIs themselves contain commas (though unlikely in practice)

Solution 1.1: Generalized Array Parsing Syntax
Approach: Implement a universal array parsing mechanism using a special syntax (e.g., ${{array:REDIRECT_URIS}}).
Implementation:

• Introduce a custom syntax that indicates a string should be parsed as an array
• Create a generic parser that handles any field marked with this syntax
• Parse the string content (comma-separated or other delimiter) into an array at runtime

Solution 2: Single Redirect URI Parameterization
Approach: Parameterize only one redirect URI per environment, assuming most environments need only one primary redirect URI.
Let users add parameters if required.

• This will break point when trying to do full automation between environments

Solution 3: Parameterize Redirect URI Count
Approach: Use the existing application structure but parameterize the number of redirect URIs, generating environment variables dynamically based on count.

In practical world redirect uri count can be different.

WDYT? @darshanasbg @senthalan

[senthalan]

Could you explain how solution 3 will work ?

• will the users can indicate how many number of redirect URIs to be present in the application in the export request ? or it will be edited by the users after ?

[rajithacharith]

Thinking of users will edit the file after getting the files, This was not fully automation friendly approach.

[senthalan]

If we can allow the users to indicate how many number of redirect URIs to be present in the application in the export request, can this be an automation friendly approach ?