|
| 1 | +--- |
| 2 | +title: Starlark in woodpecker CI |
| 3 | +authorName: Prashant Gurung |
| 4 | +authorAvatar: https://avatars.githubusercontent.com/u/53248463?v=4 |
| 5 | +authorLink: https://github.com/prashant-gurung899 |
| 6 | +createdAt: July 3, 2025 |
| 7 | +tags: CI/CD |
| 8 | +banner: https://raw.githubusercontent.com/JankariTech/blog/master/src/imgs/fallback_banner.png |
| 9 | +--- |
| 10 | + |
| 11 | +## Background |
| 12 | +Continuous Integration (CI) tools are vital for automating the testing and deployment of modern software. One such open-source tool is Woodpecker CI, a lightweight CI/CD system. Traditionally, Woodpecker pipelines are written in YAML. But with the rise of programmable pipelines, we now have the flexibility to define our CI configuration using Starlark — a Python-like configuration language. |
| 13 | + |
| 14 | +In this blog, I’ll walk you through how I set up Woodpecker CI with a dynamic Starlark-to-YAML conversion service (WCCS), enabling us to write pipelines in .drone.star files. Here's a high-level view of what we'll cover: |
| 15 | + |
| 16 | +- Spinning up a Woodpecker server |
| 17 | + |
| 18 | +- Authenticating with GitHub |
| 19 | + |
| 20 | +- Enabling a repository |
| 21 | + |
| 22 | +- Setting up WCCS (Woodpecker Config Conversion Service) |
| 23 | + |
| 24 | +- Connecting WCCS with the Woodpecker server |
| 25 | + |
| 26 | +By the end, you’ll be able to write CI pipelines in Starlark and dynamically convert them to YAML during runtime. |
| 27 | + |
| 28 | + |
| 29 | +## Setting Up Woodpecker Server |
| 30 | +We started by running the Woodpecker server using Docker. Here's the essential setup: |
| 31 | + |
| 32 | +```yml |
| 33 | +version: '3' |
| 34 | +services: |
| 35 | + woodpecker-server: |
| 36 | + image: woodpeckerci/woodpecker-server:latest |
| 37 | + ports: |
| 38 | + - 8000:8000 |
| 39 | + environment: |
| 40 | + - WOODPECKER_OPEN=true |
| 41 | + - WOODPECKER_HOST=https://<ngrok-or-your-domain> |
| 42 | + - WOODPECKER_GITHUB=true |
| 43 | + - WOODPECKER_GITHUB_CLIENT=<client-id> |
| 44 | + - WOODPECKER_GITHUB_SECRET=<client-secret> |
| 45 | + - WOODPECKER_AGENT_SECRET=supersecret |
| 46 | + - WOODPECKER_CONFIG_ENDPOINT=http://wccs:3000/ciconfig |
| 47 | + volumes: |
| 48 | + - woodpecker-data:/var/lib/woodpecker |
| 49 | + |
| 50 | + woodpecker-agent: |
| 51 | + image: woodpeckerci/woodpecker-agent:latest |
| 52 | + depends_on: |
| 53 | + - woodpecker-server |
| 54 | + environment: |
| 55 | + - WOODPECKER_SERVER=http://woodpecker-server:8000 |
| 56 | + - WOODPECKER_AGENT_SECRET=supersecret |
| 57 | + volumes: |
| 58 | + - /var/run/docker.sock:/var/run/docker.sock |
| 59 | + |
| 60 | +volumes: |
| 61 | + woodpecker-data: |
| 62 | +``` |
| 63 | +
|
| 64 | +## Authenticating with GitHub |
| 65 | +
|
| 66 | +To connect GitHub with Woodpecker: |
| 67 | +
|
| 68 | +1. Register a new OAuth application in your GitHub developer settings. |
| 69 | +
|
| 70 | +2. Set the callback URL to https://<your-ngrok-url>/authorize |
| 71 | +
|
| 72 | +3. Copy the generated Client ID and Secret. |
| 73 | +
|
| 74 | +4. Use these values in the WOODPECKER_GITHUB_CLIENT and WOODPECKER_GITHUB_SECRET env vars. |
| 75 | +
|
| 76 | +After that, you can log into the Woodpecker web UI using your GitHub account. |
| 77 | +
|
| 78 | +## Enabling a Repository |
| 79 | +From the Woodpecker UI: |
| 80 | +
|
| 81 | +- Select your GitHub repo. |
| 82 | +
|
| 83 | +- Enable it for CI. |
| 84 | +
|
| 85 | +Woodpecker will automatically add the necessary webhooks to the repo. |
| 86 | +
|
| 87 | +## Setting Up WCCS-Woodpecker CI Config Service (Starlark Conversion Service) |
| 88 | +WCCS is a small web service that receives a signed POST request from Woodpecker and returns a valid YAML pipeline. We deployed it with Docker: |
| 89 | +```yml |
| 90 | +wccs: |
| 91 | + image: opencloudeu/wccs:latest |
| 92 | + ports: |
| 93 | + - 3000:3000 |
| 94 | + command: server |
| 95 | +``` |
| 96 | +
|
| 97 | +## Generating the Public Key |
| 98 | +We started the Woodpecker server first, let it generate a key pair, and then copied the public key from the container or volume path: |
| 99 | +```console |
| 100 | +docker cp woodpecker-server:/var/lib/woodpecker/keys/public.pem ./woodpecker.pub |
| 101 | +``` |
| 102 | + |
| 103 | +## Connecting WCCS with Woodpecker |
| 104 | +To allow Woodpecker to fetch pipeline configs from WCCS, we added this to the server env: |
| 105 | +```console |
| 106 | +WOODPECKER_CONFIG_ENDPOINT=http://wccs:3000/ciconfig |
| 107 | +``` |
| 108 | +Woodpecker now sends a signed JSON payload to WCCS whenever a build is triggered. WCCS verifies the signature using the public key and responds with a YAML pipeline based on your .woodpecker.star file. |
| 109 | + |
| 110 | +## Testing WCCS Manually |
| 111 | + |
| 112 | +To debug or test WCCS manually, we used curl: |
| 113 | +```console |
| 114 | +curl -v -X POST http://localhost:3000/ciconfig \ |
| 115 | + -H "Content-Type: application/json" \ |
| 116 | + -d '{"repo": {"name": "demo"}, "build": {"event": "push", "branch": "main"}}' |
| 117 | +``` |
| 118 | + |
| 119 | +## Sample Starlark Pipeline |
| 120 | +Here's a simple .woodpecker.star example: |
| 121 | +```console |
| 122 | +def main(ctx): |
| 123 | + return { |
| 124 | + "steps": [ |
| 125 | + { |
| 126 | + "name": "hello", |
| 127 | + "image": "alpine", |
| 128 | + "commands": ["echo Hello from Starlark"] |
| 129 | + } |
| 130 | + ] |
| 131 | + } |
| 132 | +``` |
| 133 | +This gets converted by WCCS into a valid Woodpecker pipeline YAML. |
| 134 | + |
| 135 | +Now, we can write pipelines in Starlark and let WCCS handle the conversion on the fly. This approach brings flexibility, structure, and the power of logic-based configurations to our CI pipelines. |
| 136 | + |
| 137 | +Want to try it yourself? Check out Woodpecker's config service example and start building smarter pipelines today! |
0 commit comments