diff --git a/site/content/en/docs/codebase-walkthrough.md b/site/content/en/docs/codebase-walkthrough.md new file mode 100644 index 000000000..6022761f1 --- /dev/null +++ b/site/content/en/docs/codebase-walkthrough.md @@ -0,0 +1,429 @@ +--- +title: "Codebase Walkthrough" +weight: 100 +description: > + Detailed explanation of the repository structure and key components +--- + +# Codebase Walkthrough + +## Repository Structure + +The Prow repository follows a standard Go project layout with clear separation of concerns: + +``` +prow/ +├── cmd/ # Command-line tools and applications +├── pkg/ # Shared libraries and packages +├── config/ # Configuration examples +├── site/ # Documentation site +├── test/ # Test files and test data +├── hack/ # Build scripts and utilities +├── Makefile # Build automation +├── go.mod # Go module definition +├── go.sum # Go module checksums +├── README.md # Main README +├── CONTRIBUTING.md # Contribution guidelines +└── OWNERS # Code owners file +``` + +## Directory Details + +### `/cmd` - Command-Line Tools + +This directory contains all executable Prow components. Each subdirectory represents a standalone service or tool. + +#### Core Components + +**Webhook and Event Handling:** +- `hook/` - Main webhook server that processes GitHub/Gerrit events +- `webhook-server/` - Webhook server with admission control +- `ghproxy/` - GitHub API proxy with caching +- `hmac/` - HMAC signature validation + +**Job Management:** +- `prow-controller-manager/` - Main controller manager +- `plank/` - Creates and manages Pods for jobs +- `scheduler/` - Distributes jobs across clusters +- `sinker/` - Cleans up old ProwJobs and Pods +- `horologium/` - Triggers periodic jobs + +**Status and Reporting:** +- `crier/` - Reports job status to GitHub/Gerrit/Slack +- `status-reconciler/` - Reconciles GitHub commit status +- `deck/` - Web UI for viewing jobs and results +- `exporter/` - Exports metrics + +**Automation:** +- `tide/` - Automated PR merging +- `peribolos/` - GitHub organization management +- `branchprotector/` - Branch protection management + +**Job Utilities:** +- `clonerefs/` - Clones git repositories +- `initupload/` - Initializes job artifacts +- `entrypoint/` - Job entrypoint wrapper +- `sidecar/` - Sidecar container for log upload +- `gcsupload/` - Uploads artifacts to GCS +- `tot/` - Test of tests (version management) + +**Developer Tools:** +- `mkpj/` - Create ProwJob YAML +- `mkpod/` - Create Pod YAML +- `checkconfig/` - Validate Prow configuration +- `config-bootstrapper/` - Bootstrap Prow config + +**External Integrations:** +- `gerrit/` - Gerrit integration +- `jenkins-operator/` - Jenkins operator +- `pipeline/` - Tekton pipeline controller +- `gangway/` - OAuth server +- `moonraker/` - Config server + +**External Plugins:** +- `external-plugins/cherrypicker/` - Cherry-pick plugin +- `external-plugins/needs-rebase/` - Rebase checker +- `external-plugins/refresh/` - Refresh plugin + +#### Component Structure Pattern + +Each component typically follows this structure: + +``` +cmd/component-name/ +├── main.go # Entry point +├── main_test.go # Tests +├── OWNERS # Code owners (if applicable) +└── ... # Component-specific files +``` + +### `/pkg` - Shared Packages + +This directory contains reusable Go packages used across multiple components. + +#### Key Packages + +**Core API:** +- `pkg/apis/prowjobs/` - ProwJob Custom Resource definitions + - `v1/types.go` - ProwJob types + - `v1/register.go` - API registration + +**Configuration:** +- `pkg/config/` - Configuration loading and management + - `config.go` - Main config structures + - `jobs.go` - Job configuration + - `tide.go` - Tide configuration + - `plugins.go` - Plugin configuration + +**Core Components:** +- `pkg/hook/` - Webhook handling logic +- `pkg/plank/` - Pod creation and management +- `pkg/scheduler/` - Job scheduling logic +- `pkg/tide/` - PR merging logic +- `pkg/crier/` - Status reporting +- `pkg/sinker/` - Cleanup logic + +**Plugin System:** +- `pkg/plugins/` - Plugin implementations + - Individual plugin packages (approve, lgtm, etc.) + - Plugin framework + +**Git Integration:** +- `pkg/github/` - GitHub API client +- `pkg/gerrit/` - Gerrit integration +- `pkg/git/` - Git utilities +- `pkg/clonerefs/` - Repository cloning + +**Job Utilities:** +- `pkg/pod-utils/` - Pod utility functions +- `pkg/entrypoint/` - Entrypoint logic +- `pkg/initupload/` - Init upload logic +- `pkg/sidecar/` - Sidecar logic +- `pkg/gcsupload/` - GCS upload logic + +**UI Components:** +- `pkg/deck/` - Deck UI backend +- `pkg/spyglass/` - Spyglass artifact viewer + +**Utilities:** +- `pkg/util/` - General utilities +- `pkg/pjutil/` - ProwJob utilities +- `pkg/kube/` - Kubernetes client utilities +- `pkg/flagutil/` - Flag utilities +- `pkg/metrics/` - Metrics collection +- `pkg/logrusutil/` - Logging utilities + +**Specialized:** +- `pkg/repoowners/` - OWNERS file parsing +- `pkg/labels/` - Label management +- `pkg/markdown/` - Markdown processing +- `pkg/slack/` - Slack integration +- `pkg/jira/` - Jira integration +- `pkg/bugzilla/` - Bugzilla integration + +### `/config` - Configuration Examples + +Example Prow configurations: +- `prow/cluster/` - Cluster configurations +- Example job configs +- Plugin configs + +### `/site` - Documentation Site + +Hugo-based documentation site: +- Static site generation +- Documentation content +- API documentation + +### `/test` - Test Files + +- `test/integration/` - Integration test suites +- Test data files + +### `/hack` - Build Scripts + +Utility scripts for development and CI: +- `hack/make-rules/` - Makefile rules +- `hack/scripts/` - Utility scripts +- `hack/tools/` - Development tools + +## Key Files and Modules + +### Main Entry Points + +**Hook (`cmd/hook/main.go`):** +- Main webhook server +- Processes GitHub/Gerrit events +- Executes plugins +- Creates ProwJobs + +**Controller Manager (`cmd/prow-controller-manager/main.go`):** +- Runs plank and scheduler controllers +- Manages ProwJob lifecycle +- Coordinates job execution + +**Plank (`pkg/plank/`):** +- Creates Pods from ProwJobs +- Manages Pod lifecycle +- Updates ProwJob status + +**Tide (`cmd/tide/main.go`):** +- Monitors PRs for merge eligibility +- Automatically merges PRs +- Manages merge pools + +### Core Components + +**ProwJob API (`pkg/apis/prowjobs/v1/types.go`):** +```go +type ProwJob struct { + Spec ProwJobSpec + Status ProwJobStatus +} + +type ProwJobSpec struct { + Type ProwJobType // presubmit, postsubmit, periodic, batch + Job string + Refs Refs + PodSpec *corev1.PodSpec + // ... +} +``` + +**Configuration (`pkg/config/config.go`):** +- `Config` - Main configuration structure +- `JobConfig` - Job configurations +- `TideConfig` - Tide configuration +- `PluginConfig` - Plugin configuration + +**Plugin Interface (`pkg/plugins/`):** +- Plugins implement specific interfaces +- Hook loads and executes plugins +- Plugins can create comments, labels, etc. + +## Component Interactions + +### How Hook Works + +1. **Webhook Reception** (`cmd/hook/main.go`): + - Receives webhook from GitHub/Gerrit + - Validates HMAC signature + - Parses event payload + +2. **Plugin Execution** (`pkg/hook/server.go`): + - Loads plugin configuration + - Executes relevant plugins + - Plugins can create ProwJobs + +3. **ProwJob Creation** (`pkg/kube/`): + - Creates ProwJob CRD + - Controller picks up ProwJob + +### How Controller Manager Works + +1. **ProwJob Watching** (`pkg/plank/`): + - Watches for new ProwJobs + - Determines if Pod should be created + +2. **Pod Creation** (`pkg/plank/`): + - Creates Pod from ProwJob spec + - Manages Pod lifecycle + - Updates ProwJob status + +3. **Scheduling** (`pkg/scheduler/`): + - Selects cluster for job + - Distributes load across clusters + +### How Tide Works + +1. **PR Monitoring** (`pkg/tide/`): + - Queries GitHub for PRs + - Checks merge eligibility + +2. **Merge Eligibility** (`pkg/tide/`): + - All tests pass + - Required approvals present + - No merge conflicts + - Branch protection satisfied + +3. **Merging** (`pkg/tide/`): + - Merges PR when eligible + - Updates status + +### How Plugins Work + +1. **Plugin Loading** (`pkg/hook/`): + - Loads plugin configuration + - Initializes plugins + +2. **Plugin Execution** (`pkg/plugins/`): + - Hook calls plugin handlers + - Plugins process events + - Plugins can modify state + +3. **Plugin Actions**: + - Create comments + - Add/remove labels + - Create ProwJobs + - Update status + +## Key Classes and Functions + +### Hook Core + +**Main Function** (`cmd/hook/main.go:main`): +- Entry point for hook +- Sets up HTTP server +- Registers webhook handler + +**Server** (`pkg/hook/server.go`): +- `ServeHTTP()` - Handles webhook requests +- `handleWebhook()` - Processes webhook +- `handlePluginEvent()` - Executes plugins + +### Controller Core + +**Plank Controller** (`pkg/plank/`): +- `syncProwJob()` - Syncs ProwJob state +- `createPod()` - Creates Pod for job +- `updateProwJob()` - Updates ProwJob status + +**Scheduler Controller** (`pkg/scheduler/`): +- `syncProwJob()` - Schedules job to cluster +- `selectCluster()` - Selects target cluster + +### Tide Core + +**Tide Controller** (`pkg/tide/`): +- `sync()` - Syncs PR state +- `isMergeable()` - Checks merge eligibility +- `mergePRs()` - Merges eligible PRs + +### Plugin Framework + +**Plugin Interface**: +```go +type PluginClient interface { + // Plugin-specific methods +} +``` + +**Common Plugins**: +- `pkg/plugins/approve/` - Approval plugin +- `pkg/plugins/lgtm/` - LGTM plugin +- `pkg/plugins/trigger/` - Job trigger plugin + +## API Surface + +### ProwJob API + +**ProwJob Resource**: +```go +type ProwJob struct { + metav1.TypeMeta + metav1.ObjectMeta + Spec ProwJobSpec + Status ProwJobStatus +} +``` + +**Job Types**: +- `Presubmit` - Run on PRs +- `Postsubmit` - Run after merge +- `Periodic` - Run on schedule +- `Batch` - Run multiple jobs + +### Configuration API + +**Config Structure**: +```go +type Config struct { + ProwConfig ProwConfig + JobConfig JobConfig + Tide TideConfig + Plank PlankConfig + // ... +} +``` + +## Data Flow + +1. **Webhook Flow:** + - GitHub sends webhook + - Hook validates and processes + - Plugins execute + - ProwJob created + +2. **Job Execution Flow:** + - Controller reconciles ProwJob + - Plank creates Pod + - Pod executes job + - Artifacts uploaded + - Status updated + +3. **Status Reporting Flow:** + - Job completes + - Crier reads status + - Reports to GitHub/Slack + - Updates commit status + +## Testing Structure + +- **Unit Tests**: `*_test.go` files alongside source +- **Integration Tests**: `test/integration/` directory +- **Test Data**: `testdata/` directories in packages + +## Build System + +- **Makefile**: Main build automation +- **Go Modules**: Dependency management +- **Ko**: Container image builds +- **CI/CD**: Automated testing and deployment + +## Extension Points + +1. **Custom Plugins**: Add to `pkg/plugins/` +2. **Custom Controllers**: Add to controller manager +3. **Custom Reporters**: Add to `pkg/crier/reporters/` +4. **Custom Job Types**: Extend ProwJob spec + diff --git a/site/content/en/docs/faq.md b/site/content/en/docs/faq.md new file mode 100644 index 000000000..b075e1c9f --- /dev/null +++ b/site/content/en/docs/faq.md @@ -0,0 +1,275 @@ +--- +title: "FAQ" +weight: 120 +description: > + Frequently asked questions and answers +--- + +# FAQ Section + +Frequently asked questions from newcomers to the Prow repository. + +## General Questions + +### What is Prow? + +Prow is a Kubernetes-based Continuous Integration and Continuous Deployment (CI/CD) system. It provides automated testing, code review automation, and project management features for Kubernetes and other open-source projects. + +### Who maintains Prow? + +The Kubernetes SIG Testing team maintains Prow. See the [OWNERS](https://github.com/kubernetes-sigs/prow/blob/main/OWNERS) file for the list of maintainers. + +### What programming language is Prow written in? + +Primarily Go (94.7%), with some TypeScript (2.5%) for the Deck frontend. + +### How do I get started? + +1. Read the [Overview](/docs/overview/) +2. Set up your development environment +3. Try running some components +4. Explore the codebase ([Codebase Walkthrough](/docs/codebase-walkthrough/)) + +## Development Questions + +### How do I build the project? + +```bash +# Build all components +make build + +# Install to $GOPATH/bin +go install ./cmd/... + +# Build specific component +go build ./cmd/hook +``` + +### How do I run tests? + +```bash +# Run all unit tests +make test + +# Run specific package tests +go test ./pkg/hook/... + +# Run integration tests +go test ./test/integration/... +``` + +### How do I contribute? + +See the [Developing and Contributing to Prow](/docs/getting-started-develop/) guide for detailed instructions. In short: +1. Fork the repository +2. Create a feature branch +3. Make your changes +4. Test your changes +5. Create a Pull Request + +## Component Questions + +### What is hook? + +Hook is the webhook server that processes GitHub/Gerrit events and executes plugins. See the [Hook documentation](/docs/components/core/hook/). + +### What is plank? + +Plank is the controller that creates and manages Pods for ProwJobs. Note: Plank is deprecated in favor of prow-controller-manager. + +### What is Tide? + +Tide automatically merges pull requests when all requirements are met. See the [Tide documentation](/docs/components/core/tide/). + +### What is Deck? + +Deck is the web UI for viewing job status, history, and results. See the [Deck documentation](/docs/components/core/deck/). + +### What is Crier? + +Crier reports job status back to GitHub, Gerrit, Slack, etc. See the [Crier documentation](/docs/components/core/crier/). + +## Configuration Questions + +### How do I configure Prow? + +Prow requires two main config files: +- `config.yaml` - Main Prow configuration +- `plugins.yaml` - Plugin configuration + +See the [Configuration documentation](/docs/config/) for details. + +### How do I add a new job? + +Add job definition to `config.yaml`: + +```yaml +presubmits: + myorg/myrepo: + - name: my-job + spec: + containers: + - image: alpine:latest + command: ["echo", "test"] +``` + +### How do I enable a plugin? + +Add plugin to `plugins.yaml`: + +```yaml +plugins: + myorg/myrepo: + - approve + - lgtm +``` + +## Job Questions + +### What are the different job types? + +- **Presubmit**: Run on pull requests +- **Postsubmit**: Run after code is merged +- **Periodic**: Run on a schedule (cron) +- **Batch**: Run multiple jobs in parallel + +See the [ProwJobs documentation](/docs/jobs/) for more details. + +### How do I trigger a job manually? + +Use `/test ` comment on a PR. + +### How do I view job logs? + +```bash +# Get Pod name +kubectl get pods + +# View logs +kubectl logs +``` + +## Plugin Questions + +### How do plugins work? + +Plugins are Go packages that hook processes and executes based on events. See the [Plugins documentation](/docs/components/plugins/). + +### How do I create a custom plugin? + +Create a new package in `pkg/plugins/` and implement the plugin interface. See the [Developing and Contributing to Prow](/docs/getting-started-develop/) guide. + +### What plugins are available? + +See `pkg/plugins/` directory for available plugins, or check the [Plugins documentation](/docs/components/plugins/). + +## Troubleshooting Questions + +### Jobs are not running + +1. Check ProwJob status: `kubectl get prowjobs` +2. Check Pod status: `kubectl get pods` +3. Review controller logs +4. Verify configuration + +### Webhooks are not working + +1. Check hook logs +2. Verify HMAC secret +3. Check GitHub webhook configuration +4. Test webhook delivery + +### Tide is not merging + +1. Check Tide logs +2. Verify PR eligibility +3. Check required labels +4. Review branch protection + +### Deck is not loading + +1. Check Deck logs +2. Verify config path +3. Check network connectivity +4. Review browser console + +## Architecture Questions + +### How does Prow work? + +1. GitHub sends webhook +2. Hook processes webhook +3. Plugins execute +4. ProwJob created +5. Controller reconciles ProwJob +6. Plank creates Pod +7. Pod executes job +8. Status reported back + +See the [Architecture documentation](/docs/overview/architecture/) for more details. + +### How are jobs executed? + +Jobs execute as Kubernetes Pods. The controller manager creates Pods from ProwJob specs. See the [Life of a Prow Job](/docs/life-of-a-prow-job/) for details. + +### How does Tide work? + +Tide monitors PRs and automatically merges them when all requirements are met. See the [Tide documentation](/docs/components/core/tide/) for details. + +## Contribution Questions + +### How do I find good first issues? + +Look for issues labeled: +- `good first issue` +- `help wanted` +- `beginner friendly` + +### What makes a good PR? + +- Clear description +- Focused changes +- Tests included +- Documentation updated +- All checks passing + +### How long does review take? + +Typically 1-3 business days, depending on: +- PR size and complexity +- Reviewer availability +- Number of review rounds needed + +## Getting Help + +### Where can I ask questions? + +- **GitHub Issues**: For bugs and feature requests +- **Pull Requests**: For code-related questions +- **Slack**: #sig-testing on Kubernetes Slack +- **Documentation**: Check the docs in this directory + +### How do I report a bug? + +Create a GitHub issue with: +- Clear description +- Steps to reproduce +- Expected vs actual behavior +- Relevant logs/configs +- Environment information + +### How do I request a feature? + +Create a GitHub issue with: +- Problem description +- Proposed solution +- Use cases +- Any alternatives considered + +## Still Have Questions? + +- Check the [documentation index](/docs/) +- Search existing GitHub issues +- Ask in #sig-testing on Kubernetes Slack +- Create a new issue with your question + diff --git a/site/content/en/docs/onboarding.md b/site/content/en/docs/onboarding.md new file mode 100644 index 000000000..907d0cbfc --- /dev/null +++ b/site/content/en/docs/onboarding.md @@ -0,0 +1,333 @@ +--- +title: "Onboarding Guide for New Contributors" +weight: 95 +description: > + Learning path for new contributors, important concepts, and beginner roadmap +--- + +# Onboarding Guide for New Contributors + +Welcome to Prow! This guide will help you understand the codebase and get started contributing effectively. + +## What to Learn Before Contributing + +### Essential Knowledge + +1. **Go Programming Language** + - Go basics (variables, functions, structs, interfaces) + - Go concurrency (goroutines, channels) + - Go testing (`testing` package) + - Go modules and dependency management + - **Resources**: [A Tour of Go](https://tour.golang.org/), [Effective Go](https://golang.org/doc/effective_go) + +2. **Kubernetes** + - Kubernetes API concepts + - Custom Resources (CRDs) + - Pods, Services, ConfigMaps, Secrets + - Controllers and Operators + - **Resources**: [Kubernetes Documentation](https://kubernetes.io/docs/) + +3. **Git and GitHub** + - Git workflow (branching, merging, rebasing) + - GitHub API + - Webhooks + - **Resources**: [Git Documentation](https://git-scm.com/doc), [GitHub API](https://docs.github.com/en/rest) + +4. **CI/CD Concepts** + - Continuous Integration principles + - Build pipelines + - Test automation + - **Resources**: [CI/CD Best Practices](https://www.redhat.com/en/topics/devops/what-is-ci-cd) + +### Recommended Knowledge + +1. **Kubernetes Controllers** + - Controller pattern + - Reconciliation loops + - **Resources**: [Kubernetes Controller Pattern](https://kubernetes.io/docs/concepts/architecture/controller/) + +2. **Webhooks** + - Webhook concepts + - HMAC signatures + - Event handling + - **Resources**: [GitHub Webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks) + +3. **REST APIs** + - HTTP methods + - JSON handling + - Authentication + - **Resources**: [REST API Tutorial](https://restfulapi.net/) + +## Important Concepts the Repo Uses + +### 1. ProwJob Custom Resource + +Prow uses Kubernetes Custom Resources to represent CI jobs: + +```go +// From pkg/apis/prowjobs/v1/types.go +type ProwJob struct { + Spec ProwJobSpec + Status ProwJobStatus +} +``` + +**Key Concepts:** +- ProwJobs are Kubernetes resources +- Jobs execute as Pods +- Status tracks job execution + +**Learn More:** +- Study `pkg/apis/prowjobs/v1/types.go` +- Read about Kubernetes CRDs + +### 2. Controller Pattern + +Prow components use the Kubernetes controller pattern: + +```go +// Controllers watch resources and reconcile state +type Controller interface { + Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) +} +``` + +**Key Concepts:** +- Watch for resource changes +- Reconcile desired state +- Handle errors gracefully + +**Learn More:** +- Study `pkg/plank/` +- Study `pkg/scheduler/` +- Read [Kubernetes Controller Pattern](https://kubernetes.io/docs/concepts/architecture/controller/) + +### 3. Plugin System + +Prow has an extensible plugin architecture: + +```go +// Plugins implement specific interfaces +type PluginClient interface { + // Plugin methods +} +``` + +**Key Concepts:** +- Plugins are Go packages +- Hook loads and executes plugins +- Plugins can interact with GitHub, Kubernetes, etc. + +**Learn More:** +- Study `pkg/plugins/` +- Look at example plugins like `pkg/plugins/approve/` + +### 4. Webhook Handling + +Hook processes GitHub/Gerrit webhooks: + +**Key Concepts:** +- Validates HMAC signatures +- Parses event payloads +- Executes plugins +- Creates ProwJobs + +**Learn More:** +- Study `pkg/hook/server.go` +- Read about GitHub webhooks + +### 5. Job Execution + +Jobs execute as Kubernetes Pods: + +**Key Concepts:** +- Plank creates Pods from ProwJobs +- Pods run job containers +- Artifacts uploaded to GCS +- Status reported back + +**Learn More:** +- Study `pkg/plank/` +- Study `pkg/pod-utils/` + +### 6. Configuration Management + +Prow uses YAML configuration files: + +**Key Concepts:** +- Config loaded from files +- Job definitions in config +- Plugin configuration +- Tide configuration + +**Learn More:** +- Study `pkg/config/config.go` +- Look at example configs + +## Beginner Roadmap for Mastering the Project + +### Phase 1: Understanding the Basics (Week 1-2) + +**Goal**: Understand what Prow does and how it's organized + +**Tasks:** +1. Read [Overview](/docs/overview/) and [Architecture](/docs/overview/architecture/) +2. Set up development environment +3. Build and run a simple component locally +4. Read through `cmd/hook/main.go` to understand entry point +5. Study a simple component like `cmd/sinker/` + +**Deliverable**: You can build and run components locally + +### Phase 2: Understanding Hook (Week 3-4) + +**Goal**: Understand how webhooks are processed + +**Tasks:** +1. Read hook documentation +2. Study `pkg/hook/server.go` to understand webhook handling +3. Study `pkg/hook/events.go` to understand event types +4. Create a test webhook and process it locally +5. Trace through webhook processing flow + +**Deliverable**: You understand how hook works + +### Phase 3: Understanding Controllers (Week 5-6) + +**Goal**: Understand how controllers manage ProwJobs + +**Tasks:** +1. Study `pkg/plank/` to understand Pod creation +2. Study `pkg/scheduler/` to understand job scheduling +3. Study `pkg/sinker/` to understand cleanup +4. Run a controller locally and observe behavior +5. Make a small change to a controller + +**Deliverable**: You can modify controllers + +### Phase 4: Understanding Plugins (Week 7-8) + +**Goal**: Understand the plugin system + +**Tasks:** +1. Study `pkg/plugins/` to understand plugin framework +2. Study a simple plugin like `pkg/plugins/welcome/` +3. Study a complex plugin like `pkg/plugins/trigger/` +4. Create a simple custom plugin +5. Test plugin locally + +**Deliverable**: You can create custom plugins + +### Phase 5: Making Your First Contribution (Week 9-10) + +**Goal**: Make your first meaningful contribution + +**Tasks:** +1. Find a good first issue (labeled "good first issue") +2. Understand the problem and proposed solution +3. Implement the fix or feature +4. Write tests +5. Create a Pull Request +6. Address review feedback + +**Deliverable**: Your first merged PR! + +## Learning Resources by Topic + +### Go + +- **Books**: "The Go Programming Language" by Donovan & Kernighan +- **Online**: [Go by Example](https://gobyexample.com/) +- **Practice**: [Exercism Go Track](https://exercism.org/tracks/go) + +### Kubernetes + +- **Official Docs**: [Kubernetes Documentation](https://kubernetes.io/docs/) +- **Interactive**: [Kubernetes Playground](https://www.katacoda.com/courses/kubernetes) +- **Books**: "Kubernetes: Up and Running" by Hightower et al. + +### CI/CD + +- **Concepts**: [Red Hat CI/CD Guide](https://www.redhat.com/en/topics/devops/what-is-ci-cd) +- **Best Practices**: [ThoughtWorks CI/CD Guide](https://www.thoughtworks.com/continuous-integration) + +## Common Patterns in the Codebase + +### 1. Component Pattern + +Most components follow this pattern: + +```go +func main() { + // Parse flags + // Load configuration + // Initialize clients + // Start server/controller + // Handle shutdown +} +``` + +### 2. Controller Reconciliation Pattern + +```go +func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + // Fetch resource + // Check if deletion + // Reconcile desired state + // Update status + // Return result +} +``` + +### 3. Plugin Pattern + +```go +func HandleIssueComment(e github.IssueCommentEvent) error { + // Parse comment + // Execute action + // Update state + return nil +} +``` + +### 4. Error Handling Pattern + +```go +// Wrap errors with context +if err != nil { + return fmt.Errorf("operation failed: %w", err) +} + +// Use errors.Is for error checking +if errors.Is(err, os.ErrNotExist) { + // handle +} +``` + +## Tips for Success + +1. **Start Small**: Begin with small, focused changes +2. **Read Code**: Spend time reading existing code before writing new code +3. **Ask Questions**: Don't hesitate to ask for help +4. **Write Tests**: Always write tests for new code +5. **Review PRs**: Review other PRs to learn patterns +6. **Be Patient**: Understanding a large codebase takes time + +## Getting Help + +- **GitHub Issues**: Search existing issues or create new ones +- **Pull Requests**: Ask questions in PR comments +- **Slack**: #sig-testing on Kubernetes Slack +- **Documentation**: Read the docs in this directory + +## Next Steps + +After completing the roadmap: + +1. Find areas of interest +2. Look for issues labeled "good first issue" +3. Start contributing regularly +4. Consider becoming a maintainer (after significant contributions) + +Welcome to the team! 🚀 + diff --git a/site/content/en/docs/summaries.md b/site/content/en/docs/summaries.md new file mode 100644 index 000000000..b6a50a780 --- /dev/null +++ b/site/content/en/docs/summaries.md @@ -0,0 +1,236 @@ +--- +title: "Summaries" +weight: 110 +description: > + Summaries at different technical levels (non-technical, intermediate, advanced) +--- + +# Summaries + +This document provides summaries at different technical levels for different audiences. + +## Non-Technical Summary + +**What is Prow?** + +Prow is a software system that helps developers automatically test their code and manage software projects. Think of it as an automated assistant for software development teams. + +**What does it do?** + +When developers write code and submit it for review, Prow automatically: +- Runs tests to make sure the code works correctly +- Checks if the code follows project rules +- Helps manage the review process +- Automatically merges code when it's ready + +**Why is it important?** + +Large software projects like Kubernetes have thousands of developers submitting code every day. Without Prow, managing all the tests and reviews would be extremely time-consuming and error-prone. Prow automates these tasks, making the development process faster, more reliable, and consistent. + +**Who uses it?** + +Primarily the Kubernetes project and other large open-source projects. It runs automatically in the background, handling thousands of code reviews and tests every day. + +**Key Benefits:** +- Saves time by automating repetitive tasks +- Reduces errors through consistent processes +- Enables faster software releases +- Provides visibility into code quality and test status + +## Intermediate Summary + +**What is Prow?** + +Prow is a Kubernetes-based Continuous Integration (CI) and Continuous Deployment (CD) system. It provides automated testing, code review automation, and project management features for large-scale software projects. + +**Core Components:** + +1. **Hook**: Webhook server that processes GitHub/Gerrit events and executes plugins based on those events. It validates webhook signatures and creates ProwJobs for CI tasks. + +2. **Controller Manager**: Runs controllers (plank, scheduler) that manage ProwJobs. Plank creates Kubernetes Pods from ProwJobs, while scheduler distributes jobs across multiple clusters. + +3. **Tide**: Automated PR merging system that monitors pull requests and merges them when all requirements are met (tests pass, approvals received, etc.). + +4. **Deck**: Web UI that provides visibility into job status, history, and results. It integrates with Spyglass for viewing artifacts. + +5. **Crier**: Status reporting component that reports job results back to GitHub, Gerrit, Slack, and other systems. + +6. **Plugins**: Extensible plugin system that automates various tasks like labeling PRs, managing approvals, triggering tests, etc. + +**How it Works:** + +1. Developer opens a pull request on GitHub +2. GitHub sends webhook to Prow's hook component +3. Hook validates webhook and executes relevant plugins +4. Plugins create ProwJobs for CI tasks +5. Controller manager reconciles ProwJobs +6. Plank creates Kubernetes Pods to run jobs +7. Jobs execute and upload artifacts +8. Crier reports results back to GitHub +9. Tide merges PR if all requirements are met + +**Key Features:** + +- **Kubernetes Native**: Uses Kubernetes Custom Resources (ProwJobs) and Pods +- **Plugin Architecture**: Extensible plugin system for automation +- **Multi-Cluster**: Can distribute jobs across multiple Kubernetes clusters +- **Scalability**: Handles thousands of repositories and millions of test runs +- **Integration**: Deep integration with GitHub, Gerrit, Slack, etc. + +**Technology Stack:** + +- Go for backend components +- Kubernetes for orchestration +- TypeScript/React for Deck UI +- YAML for configuration + +**Use Cases:** + +- Automated testing on pull requests +- Code review automation +- Project management (labeling, milestones) +- Automated PR merging +- Status reporting and notifications + +## Advanced Summary + +**What is Prow?** + +Prow is a comprehensive Kubernetes-native CI/CD platform implementing a declarative, event-driven architecture for automated testing, code review, and project management at scale. It uses Kubernetes Custom Resources (ProwJobs) to represent CI jobs and executes them as Pods. + +**Architecture:** + +The system follows a microservices architecture where independent components communicate through: +- **Kubernetes API**: ProwJobs, Pods, ConfigMaps, Secrets +- **GitHub/Gerrit API**: Webhooks, status updates, PR management +- **Event-Driven**: Responds to webhooks, cron schedules, and resource changes + +**Core Design Patterns:** + +1. **Kubernetes Controller Pattern**: Components like plank and scheduler implement the controller pattern: + - Watch ProwJobs via informers + - Reconcile desired state + - Handle errors with exponential backoff + - Update resource status + +2. **Plugin Architecture**: Extensible plugin system: + - Plugins are Go packages in `pkg/plugins/` + - Hook loads plugins dynamically + - Plugins implement specific interfaces + - Plugins can create ProwJobs, modify GitHub state, etc. + +3. **Custom Resources**: ProwJobs are Kubernetes CRDs: + ```go + type ProwJob struct { + Spec ProwJobSpec // Job definition + Status ProwJobStatus // Execution status + } + ``` + - Jobs execute as Pods + - Status tracks execution state + - Controllers manage lifecycle + +4. **Webhook Processing**: Hook component: + - Validates HMAC signatures + - Parses event payloads + - Executes plugins based on event type + - Creates ProwJobs for CI tasks + +5. **Multi-Cluster Scheduling**: Scheduler distributes jobs: + - Selects cluster based on job requirements + - Load balances across clusters + - Handles cluster-specific configurations + +**Key Technical Components:** + +1. **Hook** (`cmd/hook/`, `pkg/hook/`): + - HTTP server processing webhooks + - Plugin execution engine + - ProwJob creation logic + - HMAC validation + +2. **Controller Manager** (`cmd/prow-controller-manager/`): + - Runs multiple controllers (plank, scheduler) + - Uses controller-runtime framework + - Manages ProwJob lifecycle + +3. **Plank** (`pkg/plank/`): + - Creates Pods from ProwJob specs + - Manages Pod lifecycle + - Updates ProwJob status + - Handles job completion + +4. **Scheduler** (`pkg/scheduler/`): + - Selects target cluster for jobs + - Distributes load across clusters + - Handles cluster-specific configs + +5. **Tide** (`cmd/tide/`, `pkg/tide/`): + - Queries GitHub for PRs + - Checks merge eligibility + - Manages merge pools + - Automatically merges PRs + +6. **Plugin System** (`pkg/plugins/`): + - Plugin framework + - Individual plugin implementations + - Plugin configuration management + +**Advanced Features:** + +1. **In-Repo Configuration**: Support for configuration in repository +2. **Gerrit Integration**: Full support for Gerrit code review +3. **Tekton Integration**: Support for Tekton pipelines +4. **Spyglass**: Unified artifact viewer +5. **Gangway**: OAuth server for Prow access + +**Scalability Considerations:** + +- **Horizontal Scaling**: Components scale horizontally +- **Multi-Cluster**: Jobs distributed across clusters +- **Caching**: ghproxy caches GitHub API responses +- **Resource Management**: Sinker cleans up old resources +- **Efficient API Usage**: Informers and watches for Kubernetes API + +**Integration Points:** + +- **GitHub API**: Webhooks, status updates, PR management +- **Kubernetes API**: Resource management, event watching +- **Gerrit API**: Code review integration +- **Slack API**: Notifications +- **GCS**: Artifact storage +- **Pub/Sub**: Event streaming + +**Extension Mechanisms:** + +1. **Custom Plugins**: Implement plugin interface in `pkg/plugins/` +2. **Custom Controllers**: Add to controller manager +3. **Custom Reporters**: Add to `pkg/crier/reporters/` +4. **Custom Job Types**: Extend ProwJob spec + +**Performance Optimizations:** + +- Informer-based resource watching +- Efficient GitHub API usage with caching +- Parallel job execution +- Resource cleanup via sinker +- Multi-cluster load distribution + +**Security Model:** + +- HMAC signature validation for webhooks +- RBAC for Kubernetes resource access +- OWNERS files for GitHub permissions +- Service account-based authentication +- Secret management via Kubernetes Secrets + +**Monitoring and Observability:** + +- Prometheus metrics for all components +- Structured logging with logrus +- OpenTelemetry tracing (where applicable) +- Deck UI for job visibility +- Custom dashboards for key metrics + +This architecture enables Prow to handle the scale and complexity of large projects like Kubernetes while remaining maintainable and extensible. +