[Design] Rework our CI integration

[Freyskeyd]

Refactor CI/CD setup across 3+ repositories to share common configuration, workflows, and processes while maintaining consistency in PR flows, merge group handling, and release management.

Goals

1. Create centralized reusable workflow repository
2. Standardize PR templates and repository settings
3. Implement consistent changelog generation
4. Establish shared CI/CD patterns for all project stages

Implementation Plan

Phase 1: Reusable Workflows (7 Iterations)

Iteration 1: Pending PR - PR validation, code quality, basic security
Iteration 2: Reviewed PR - Advanced testing, performance, coverage
Iteration 3: PR in merge-group - Pre-merge validation, conflict checks, final tests
Iteration 4: Commit to main - Post-merge validation, deployment, artifacts
Iteration 5: Daily tidying - Dependency updates, daily security, health checks
Iteration 6: Weekly tidying - Comprehensive audits, performance analysis, cleanup
Iteration 7: Release processes - Version management, changelog, publishing

Phase 2: Repository Standardization

• Enhanced PR templates with breaking changes, config changes, checklists
• Repository settings management (see REPOSITORY_CONFIGURATION.md)
• Branch protection rules and security policies

Phase 3: Migration and Validation

• Migrate repositories to use reusable workflows
• Validate CI/CD functionality across all repositories
• Implement monitoring and documentation

Tasks by Phase

Phase 1: Reusable Workflows

• Create agglayer/reusable-workflows repository
• Implement workflows for each iteration (1-7)
• Create composite actions for common setup steps
• Document workflow inputs and outputs

Phase 2: Repository Standardization

• Choose configuration management approach
• Create enhanced PR and issue templates
• Implement PR title validation and automated labelling

Phase 3: Migration and Validation

• Audit current workflows in each repository
• Migrate repositories to use reusable workflows
• Setup monitoring, notifications, and compliance checks
• Create validation scripts and documentation

Success Criteria

1. Consistency: All repositories use identical CI/CD patterns
2. Maintainability: Changes to reusable workflows propagate automatically
3. Reliability: CI/CD success rate > 95%
4. Speed: Pipeline completion time < 15 minutes
5. Security: Consistent security scanning across all repositories

Timeline

• Weeks 1-14: Phase 1 - Reusable Workflows (2 weeks per iteration)
• Weeks 15-16: Phase 2 - Repository Standardization
• Weeks 17-20: Phase 3 - Migration and Validation