Skip to content

Commit 485ca62

Browse files
add blog on how to use starlark in Woodpecker CI
Signed-off-by: prashant-gurung899 <prasantgrg777@gmail.com>
1 parent 0c1007c commit 485ca62

File tree

4 files changed

+234
-0
lines changed

4 files changed

+234
-0
lines changed
49.3 KB
Loading
37.9 KB
Loading
37.7 KB
Loading
Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
---
2+
title: Starlark in woodpecker CI
3+
authorName: Prashant Gurung
4+
authorAvatar: https://avatars.githubusercontent.com/u/53248463?v=4
5+
authorLink: https://github.com/prashant-gurung899
6+
createdAt: July 3, 2025
7+
tags: CI/CD
8+
banner: https://raw.githubusercontent.com/JankariTech/blog/woodpecker-ci-starlark/src/assets/woodpeckerCI/images/woodpecker.png
9+
---
10+
11+
## Background
12+
Continuous Integration (CI) tools are vital for automating the testing and deployment of modern software. One such open-source tool is Woodpecker CI, a lightweight CI/CD system. Woodpecker CI is an open-source continuous integration and delivery platform that helps developers automate building, testing, and deploying their code. It is a community-driven fork and successor of the popular Drone CI project, sharing many design principles and compatibility with Drone pipelines. Traditionally, Woodpecker pipelines are written in YAML. But with the rise of programmable pipelines, we now have the flexibility to define our CI configuration using Starlark — a Python-like configuration language.
13+
14+
In this blog, I’ll walk you through setting up Woodpecker CI with Traefik as a reverse proxy and integrating it with WCCS (Woodpecker Config Service) to convert Starlark configurations into YAML files, enabling us to write pipelines in .woodpecker.star files. Here's a high-level view of what we'll cover:
15+
16+
- Spinning up a Woodpecker server
17+
18+
- Authenticating with GitHub
19+
20+
- Enabling a repository
21+
22+
- Connecting WCCS with the Woodpecker server
23+
24+
By the end, you’ll be able to write CI pipelines in Starlark and dynamically convert them to YAML during runtime.
25+
26+
## Prepare Woodpecker secrets via .env
27+
```console
28+
Your woodpecker host
29+
WOODPECKER_HOST=https://<your-ci-server>
30+
31+
32+
# GitHub OAuth Secrets
33+
WOODPECKER_GITHUB_CLIENT=XXXXXXXXXXXXXXXX
34+
WOODPECKER_GITHUB_SECRET=XXXXXXXXXXXXXXXXXXXXXXXX
35+
36+
37+
# Shared secret used by server<->agent - you can generate it using openSSL
38+
WOODPECKER_AGENT_SECRET=<generated-secret>
39+
40+
41+
# Let’s Encrypt contact email (Traefik will use this)
42+
ACME_EMAIL=admin@example.com
43+
```
44+
45+
## Traefik Configuration
46+
Traefik plays a crucial role in this architecture by handling all the networking complexities - SSL termination, routing, and load balancing
47+
48+
```yml
49+
version: "3.9"
50+
51+
services:
52+
traefik:
53+
image: traefik:v3.1
54+
container_name: traefik
55+
command:
56+
- "--log.level=DEBUG"
57+
- "--api.insecure=true"
58+
- "--providers.docker=true"
59+
- "--providers.docker.exposedbydefault=false"
60+
- "--entrypoints.web.address=:80"
61+
- "--entrypoints.websecure.address=:443"
62+
- "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
63+
- "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
64+
- "--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL}"
65+
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
66+
ports:
67+
- "80:80"
68+
- "443:443"
69+
volumes:
70+
- "./letsencrypt:/letsencrypt"
71+
- "/var/run/docker.sock:/var/run/docker.sock:ro"
72+
networks:
73+
- woodpecker-net
74+
```
75+
76+
This configuration:
77+
78+
- Sets up Traefik to listen on ports 80 and 443
79+
- Configures Let's Encrypt for automatic SSL certificate management
80+
- Enables Docker provider to automatically detect new services
81+
- Mounts necessary volumes for certificate storage and Docker socket access
82+
83+
## Setting Up Woodpecker Server and Agent
84+
Now that we have Traefik running, let's set up the Woodpecker server and agent services. Here's the essential setup:
85+
86+
```yml
87+
woodpecker-server:
88+
depends_on:
89+
- wccs
90+
image: woodpeckerci/woodpecker-server:v3
91+
environment:
92+
- WOODPECKER_LOG_LEVEL=debug
93+
- WOODPECKER_OPEN=false
94+
- WOODPECKER_HOST=${WOODPECKER_HOST}
95+
- WOODPECKER_GITHUB=true
96+
- WOODPECKER_GITHUB_CLIENT=${WOODPECKER_GITHUB_CLIENT}
97+
- WOODPECKER_GITHUB_SECRET=${WOODPECKER_GITHUB_SECRET}
98+
- WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
99+
- WOODPECKER_ADMIN=prashant-gurung899
100+
- WOODPECKER_CONFIG_SERVICE_ENDPOINT=http://wccs:8080/ciconfig
101+
volumes:
102+
- woodpecker-server-data:/var/lib/woodpecker/
103+
labels:
104+
- "traefik.enable=true"
105+
- "traefik.http.routers.woodpecker-secure.rule=Host(`your-ci-server`)"
106+
- "traefik.http.routers.woodpecker-secure.entrypoints=websecure"
107+
- "traefik.http.routers.woodpecker-secure.tls.certresolver=letsencrypt"
108+
- "traefik.http.routers.woodpecker-secure.tls=true"
109+
- "traefik.http.services.woodpecker-secure.loadbalancer.server.port=8000"
110+
# HTTP router + redirect to HTTPS
111+
- "traefik.http.routers.woodpecker-http.rule=Host(`your-ci-server`)"
112+
- "traefik.http.routers.woodpecker-http.entrypoints=web"
113+
- "traefik.http.routers.woodpecker-http.middlewares=redirect-to-https"
114+
# Redirect middleware
115+
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
116+
networks:
117+
- woodpecker-net
118+
119+
woodpecker-agent:
120+
image: woodpeckerci/woodpecker-agent:v3
121+
command: agent
122+
restart: always
123+
depends_on:
124+
- woodpecker-server
125+
volumes:
126+
- woodpecker-agent-config:/etc/woodpecker
127+
- /var/run/docker.sock:/var/run/docker.sock
128+
environment:
129+
- WOODPECKER_LOG_LEVEL=debug
130+
- WOODPECKER_SERVER=woodpecker-server:9000
131+
- WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
132+
networks:
133+
- woodpecker-net
134+
```
135+
136+
## Authenticating with GitHub
137+
138+
To connect GitHub with Woodpecker:
139+
140+
1. Register a new OAuth application in your GitHub developer settings.
141+
142+
2. Set the callback URL to `https://<your-ci-server>/authorize`
143+
144+
3. Copy the generated Client ID and Secret.
145+
146+
4. Use these values in the `WOODPECKER_GITHUB_CLIENT` and `WOODPECKER_GITHUB_SECRET` env vars.
147+
148+
After that, you can log into the Woodpecker web UI using your GitHub account.
149+
150+
## Enabling a Repository
151+
From the Woodpecker UI:
152+
153+
- Select your GitHub repo.
154+
![Add Repo](https://raw.githubusercontent.com/JankariTech/blog/woodpecker-ci-starlark/src/assets/woopeckerCI/images/add.png)
155+
156+
- Enable it for CI.
157+
![Enable Repo](https://raw.githubusercontent.com/JankariTech/blog/woodpecker-ci-starlark/src/assets/woopeckerCI/images/enable.png)
158+
159+
Woodpecker will automatically add the necessary webhooks to the repo.
160+
161+
162+
## Generating the Public Key
163+
Every request sent by Woodpecker is signed using a http-signature by a private key (ed25519) generated on the first start of the Woodpecker server. You can get the public key for the verification of the http-signature from:
164+
```console
165+
http(s)://your-ci-server/api/signature/public-key
166+
```
167+
Store that public key into `keys/public.pem` in the same directory level as your docker compose, it will be needed for signature verification.
168+
169+
## Setting Up WCCS-Woodpecker CI Config Service (Starlark Conversion Service)
170+
The Woodpecker Config Conversion Service (WCCS) is a lightweight web service created and maintained by [Opencloud-eu](https://opencloud.eu/en). It enables Woodpecker CI to convert pipeline definitions written in Starlark into standard YAML on the fly by receiving a signed POST request from Woodpecker.
171+
You can easily deploy WCCS using their official Docker image available on Docker Hub: [opencloudeu/wccs](https://hub.docker.com/r/opencloudeu/wccs).
172+
```yml
173+
wccs:
174+
image: opencloudeu/wccs:latest
175+
container_name: wccs
176+
command: server
177+
environment:
178+
- WCCS_LOG_LEVEL=debug
179+
- WCCS_SERVER_PUBLIC_KEY=/keys/public.pem
180+
volumes:
181+
- /opt/woodpecker/keys:/keys
182+
- /etc/ssl/certs:/etc/ssl/certs:ro
183+
labels:
184+
- "traefik.enable=true"
185+
- "traefik.http.routers.wccs.rule=Host(`your-wccs-server`)"
186+
- "traefik.http.routers.wccs.entrypoints=websecure"
187+
- "traefik.http.routers.wccs.tls.certresolver=letsencrypt"
188+
- "traefik.http.services.wccs.loadbalancer.server.port=8080"
189+
networks:
190+
- woodpecker-net
191+
192+
volumes:
193+
woodpecker-server-data:
194+
woodpecker-agent-config:
195+
196+
networks:
197+
woodpecker-net:
198+
driver: bridge
199+
```
200+
201+
## Connecting WCCS with Woodpecker
202+
To allow Woodpecker to fetch pipeline configs from WCCS, we added this to the server env:
203+
```console
204+
WOODPECKER_CONFIG_SERVICE_ENDPOINT=http://wccs:8080/ciconfig
205+
```
206+
Woodpecker now sends a signed JSON payload to WCCS whenever a build is triggered. WCCS verifies the signature using the public key and responds with a YAML pipeline based on your .woodpecker.star file.
207+
208+
## Sample Starlark Pipeline
209+
Here's a simple .woodpecker.star example:
210+
```console
211+
def main(ctx):
212+
return [{
213+
"name": "hello",
214+
"steps": [
215+
{
216+
"name": "greeting",
217+
"image": "alpine",
218+
"commands": [
219+
"echo Hello from CI",
220+
],
221+
"when": {
222+
"event": ["push", "pull_request"],
223+
"branch": ["master"],
224+
},
225+
}
226+
]
227+
}]
228+
```
229+
This gets converted by WCCS into a valid Woodpecker pipeline YAML.
230+
231+
## Conclusion
232+
This setup provides a robust, secure, and flexible CI/CD pipeline using Woodpecker CI, enhanced with WCCS for configuration management and protected by Traefik. The integration of these components creates a powerful system that can handle complex pipeline configurations while maintaining security and ease of use.
233+
234+
Now, we can write pipelines in Starlark and let WCCS handle the conversion on the fly. This approach brings flexibility, structure, and the power of logic-based configurations to our CI pipelines.

0 commit comments

Comments
 (0)