[Policy Hub] Policy Contribution Flow

[DakshithaS]

Overview

This discussion outlines the Policy Contribution Flow for the Policy Hub, detailing how policies are contributed, validated, and published through an automated CI/CD pipeline.

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub Repo (Policy Hub)
    participant Maint as WSO2 Maintainers
    participant WF as GitHub Workflows
    participant S3 as Artifact Storage
    participant Hub as Policy Hub

    Note over Dev: Developer creates a new policy locally

    Dev->>Dev: Implement policy + run local validation

    alt Local validation passes
        Dev->>GH: Submit changes (open PR)
    else Validation fails
        Dev->>Dev: Fix issues and retry
    end

    GH-->>Maint: Notify maintainers of new PR
    Maint->>GH: Review PR and validate quality

    alt PR approved
        Maint->>GH: Merge PR into main branch
    else PR rejected
        GH-->>Dev: PR closed without merging
    end

    Note over GH,WF: After merge, automation begins

    GH->>WF: Trigger GitHub CI workflows
    WF->>WF: Detect new policy versions
    WF->>WF: Validate merged policy version

    alt Validation succeeds
        WF->>S3: Upload policy artifact
        S3-->>WF: Artifact stored successfully

        WF->>Hub: Register policy in Policy Hub
        Hub-->>WF: Policy successfully added

        WF->>GH: ✓ Workflow successful
    else Validation fails
        WF->>GH: ✗ Workflow failed
        GH-->>Maint: Notify maintainers of validation failure
    end

Loading

🎯 Current Architecture

Repository Structure

policy-hub/
├── policies/           # Policy definitions
│   └── {policy-name}/
│       └── v{version}/
│           ├── metadata.json
│           ├── policy-definition.yaml
│           ├── src/main.go
│           └── docs/
├── scripts/            # Validation & utility scripts
├── config/             # Configuration files
└── .github/            # CI/CD workflows & actions

🔄 Policy Contribution Flow

Phase 1: Policy Development

Contributor Actions:

1. Fork Repository - Create fork for policy development
2. Create Policy Structure - Follow required directory layout:

   policies/{policy-name}/v{version}/
   ├── metadata.json          # Policy metadata
   ├── policy-definition.yaml # Policy rules
   ├── src/main.go           # Implementation
   └── docs/                  # Documentation
       ├── overview.md
       ├── configuration.md
       └── examples.md
   

3. Implement Policy Logic - Write Go code with proper error handling
4. Add Documentation - Complete all required documentation files
5. Test Locally - Use validation scripts:

   ./scripts/validate-policy.sh {policy-name} v{version}

Phase 2: Pull Request Submission

Automated Validation Triggers:

• PR opened
• Path filters: policies/**

Validation Steps:

1. Code Checkout - Full git history for SHA comparison
2. Policy Structure Validation
   • Required files check (metadata.json, policy-definition.yaml)
   • Required directories check (src/, docs/)
   • Documentation completeness
   • Metadata field validation
   • Go compilation validation (optional)
3. Version Detection - Identify new policy versions added
4. PR Comment - Detailed validation results with next steps

Phase 3: Pre-Release Validation

Release Trigger:

• GitHub Release published with tag (e.g., v1.2.3)

🌟 USER INTERACTION REQUIRED: To publish a policy, users need to create a tag and do a release

Batch Release Workflow (batch-release.yml):

name: Batch Release
on:
  release:
    types: [published]

Release Process:

1. Version Detection
   • Compare current release SHA with previous release tag
   • Identify all new policy versions since last release
2. Pre-Release Validation
   • Run full validation on all new policies
   • Block release if any validation fails
3. Parallel Publishing
   • Matrix strategy for concurrent publishing
   • Publish to Artifact Storage and Policy Hub API
   • Support for multiple storage backends

Discussion Points

1. Contributor Experience - How can we improve the contribution process?
2. Security Considerations - Additional security validations needed?
3. Error Recovery - How can we implement error recovery if some release goes wrong? Half fail? Rollback options?

[nuwand]

Workflow looks good to me.

[renuka-fernando]

Do we actually need GitHub releases for this, given that the main branch is always production-ready? Also, what does a GitHub tag like v1.0.12 really represent in this context—major, minor, or patch? If we add a new policy or fix a bug, should that be considered a patch-level update in the policies repository?

[DakshithaS]

1. The purpose of having a GitHub Release step is to provide users with a manual interaction point in the release process.
   Without this step, the entire flow could be fully automated, so Policy Hub release could run immediately after a PR is merged.
   However, felt it would be better to allow an additional manual checkpoint for users to trigger the release.

2. Policies will be treated as immutable.
   Once a policy version is released, no further modifications will be allowed on that same version.
   If a bug or issue is found, it must be addressed by releasing a new patch version instead.