Skip to content

Commit ea94756

Browse files
danielgerlagdanielgerlag
committed
github example
1 parent 94afd11 commit ea94756

13 files changed

Lines changed: 1296 additions & 0 deletions

File tree

‎examples/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ Comprehensive example demonstrating advanced features and production patterns.
4646

4747
---
4848

49+
### 🐙 [github-webhooks/](github-webhooks/)
50+
**GitHub Integration** - Receive and process GitHub webhook events in real-time using the HTTP source in webhook mode.
51+
52+
**Features:**
53+
- HTTP source with webhook mode and HMAC-SHA256 signature verification
54+
- Conditional mappings for push, pull request, and issue events
55+
- Continuous queries over GitHub activity
56+
- SSE streaming and console logging
57+
- Simulation scripts for local testing (no GitHub setup required)
58+
59+
**Use this for:** Learning webhook mode, building real-world HTTP integrations
60+
61+
---
62+
4963
## Quick Start
5064

5165
Each example includes:

‎examples/github-webhooks/.env‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# GitHub Webhooks Example - Default Environment Variables
2+
# For local development and testing with simulation scripts.
3+
4+
GITHUB_WEBHOOK_SECRET=my-webhook-secret
5+
SERVER_HOST=0.0.0.0
6+
SERVER_PORT=8080
7+
LOG_LEVEL=info
8+
WEBHOOK_PORT=9000
9+
SSE_HOST=0.0.0.0
10+
SSE_PORT=8081

‎examples/github-webhooks/.env.example‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# GitHub Webhooks Example - Environment Variables
2+
#
3+
# Copy this file to .env and customize:
4+
# cp .env.example .env
5+
6+
# GitHub Webhook Secret
7+
# This must match the secret configured in your GitHub repository webhook settings.
8+
# For local testing with the simulation scripts, any value works.
9+
GITHUB_WEBHOOK_SECRET=my-webhook-secret
10+
11+
# Server settings
12+
SERVER_HOST=0.0.0.0
13+
SERVER_PORT=8080
14+
LOG_LEVEL=info
15+
16+
# HTTP source port (where GitHub sends webhooks)
17+
WEBHOOK_PORT=9000
18+
19+
# SSE reaction port (for browser streaming)
20+
SSE_HOST=0.0.0.0
21+
SSE_PORT=8081

‎examples/github-webhooks/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Local environment overrides
2+
.env.local

‎examples/github-webhooks/README.md‎

Lines changed: 338 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,338 @@
1+
# GitHub Webhooks with Drasi Server
2+
3+
This example demonstrates how to use Drasi Server's HTTP source in **webhook mode** to receive and process GitHub webhook events in real-time. It showcases a practical integration scenario where continuous queries react to repository activity — pushes, pull requests, and issues.
4+
5+
## What You'll Build
6+
7+
A real-time GitHub activity monitoring system that:
8+
- Receives webhook events from GitHub via HTTP
9+
- Verifies payload signatures using HMAC-SHA256
10+
- Transforms webhook payloads into graph nodes using configurable mappings
11+
- Runs continuous Cypher queries over GitHub activity
12+
- Streams results to the browser via SSE and logs to the console
13+
14+
## Architecture
15+
16+
```
17+
┌─────────────────┐ ┌──────────────────────────────────────┐ ┌──────────────────┐
18+
│ GitHub / Curl │────▶│ Drasi Server │────▶│ Browser (SSE) │
19+
│ (Webhooks) │POST │ │ SSE │ viewer/index │
20+
│ │ │ Source: │ │ .html │
21+
│ Events: │ │ - github-webhooks (HTTP webhook) │ └──────────────────┘
22+
│ • push │ │ │
23+
│ • pull_request │ │ Queries: │ ┌──────────────────┐
24+
│ • issues │ │ - all-activity │────▶│ Console (Log) │
25+
│ │ │ - recent-pushes │ Log │ reaction │
26+
└─────────────────┘ │ - open-pull-requests │ └──────────────────┘
27+
│ - issues-opened │
28+
│ │ ┌──────────────────┐
29+
│ Reactions: │ │ REST API │
30+
│ - log-github-activity │────▶│ port 8080 │
31+
│ - sse-github-events │ └──────────────────┘
32+
└──────────────────────────────────────┘
33+
```
34+
35+
### Data Flow
36+
37+
1. **GitHub sends a webhook** (or you simulate one with the provided scripts)
38+
2. **HTTP source verifies** the HMAC-SHA256 signature against your secret
39+
3. **Webhook mappings** route different event types (push, PR, issue) based on the `X-GitHub-Event` header
40+
4. **Templates** extract fields from the JSON payload and create graph nodes with appropriate labels and properties
41+
5. **Continuous queries** filter and project the data
42+
6. **Reactions** deliver results to the console log and SSE stream
43+
44+
## Prerequisites
45+
46+
- **Rust/Cargo**: Required to build Drasi Server — [Install Rust](https://rustup.rs/)
47+
- **curl**: For running simulation scripts
48+
- **openssl**: For HMAC signature computation in simulation scripts
49+
50+
## Quick Start
51+
52+
### 1. Start the Server
53+
54+
```bash
55+
cd scripts
56+
./start-server.sh
57+
```
58+
59+
This builds and starts Drasi Server with the webhook configuration. The server will:
60+
- Listen for API requests on port **8080**
61+
- Listen for GitHub webhooks on port **9000** at `/github/events`
62+
- Stream SSE events on port **8081** at `/events`
63+
64+
### 2. Simulate GitHub Events
65+
66+
In a separate terminal, send simulated webhook events:
67+
68+
```bash
69+
# Simulate a push event
70+
./scripts/simulate-push.sh
71+
72+
# Simulate a pull request being opened
73+
./scripts/simulate-pr.sh
74+
75+
# Simulate an issue being opened
76+
./scripts/simulate-issue.sh
77+
```
78+
79+
Each script sends a realistic GitHub webhook payload with a valid HMAC-SHA256 signature.
80+
81+
### 3. View Results
82+
83+
Check query results via the REST API:
84+
85+
```bash
86+
./scripts/view-results.sh
87+
```
88+
89+
Or stream events in real-time:
90+
91+
```bash
92+
./scripts/stream-events.sh
93+
```
94+
95+
### 4. Open the Live Viewer
96+
97+
Open `viewer/index.html` in your browser to see a live dashboard of GitHub events streaming via SSE.
98+
99+
## Connecting to a Real GitHub Repository
100+
101+
To receive real GitHub webhooks you need a public URL that GitHub can reach. This section walks through using [Microsoft Dev Tunnels](https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/overview) to expose your local webhook endpoint.
102+
103+
### 1. Install the Dev Tunnels CLI
104+
105+
If you don't already have the `devtunnel` CLI:
106+
107+
```bash
108+
# macOS
109+
brew install --cask devtunnel
110+
111+
# Linux (Debian/Ubuntu)
112+
curl -sL https://aka.ms/DevTunnelCliInstall | bash
113+
114+
# Windows (winget)
115+
winget install Microsoft.devtunnel
116+
```
117+
118+
Verify the installation:
119+
120+
```bash
121+
devtunnel --version
122+
```
123+
124+
### 2. Log in
125+
126+
Authenticate with your Microsoft or GitHub account:
127+
128+
```bash
129+
# Log in with GitHub (recommended — works with any GitHub account)
130+
devtunnel user login --github
131+
132+
# Or with a Microsoft account
133+
devtunnel user login
134+
```
135+
136+
### 3. Create and start the tunnel
137+
138+
Expose the webhook port (9000 by default) with anonymous access so GitHub can reach it without additional auth headers:
139+
140+
```bash
141+
devtunnel host -p 9000 --allow-anonymous
142+
```
143+
144+
The CLI will output a public URL like:
145+
146+
```
147+
Connect via browser: https://abc123-9000.usw2.devtunnels.ms
148+
```
149+
150+
> **Tip:** To run the tunnel in the background and give it a persistent name:
151+
> ```bash
152+
> devtunnel create --name github-webhooks
153+
> devtunnel port create github-webhooks --port-number 9000
154+
> devtunnel access create github-webhooks --port-number 9000 --anonymous
155+
> devtunnel host github-webhooks
156+
> ```
157+
158+
### 4. Generate a webhook secret
159+
160+
For real usage, generate a strong secret:
161+
162+
```bash
163+
# Generate a random 32-byte hex secret
164+
openssl rand -hex 32
165+
```
166+
167+
Update your `.env` file with the generated secret:
168+
169+
```bash
170+
GITHUB_WEBHOOK_SECRET=<your-generated-secret>
171+
```
172+
173+
### 5. Configure the GitHub Webhook
174+
175+
1. Go to your repository on GitHub → **Settings** → **Webhooks** → **Add webhook**
176+
2. Set:
177+
- **Payload URL**: `https://<your-tunnel-id>-9000.usw2.devtunnels.ms/github/events`
178+
(use the URL from step 3, appending `/github/events`)
179+
- **Content type**: `application/json`
180+
- **Secret**: The secret you generated in step 4
181+
- **Events**: Select "Send me everything" or choose specific events:
182+
- ✅ Pushes
183+
- ✅ Pull requests
184+
- ✅ Issues
185+
3. Click **Add webhook**
186+
187+
GitHub will immediately send a `ping` event. You should see a green checkmark (✓) next to the webhook once Drasi Server responds with 200 OK.
188+
189+
### 6. Start Drasi Server
190+
191+
Make sure your `.env` has the correct secret, then start the server:
192+
193+
```bash
194+
cd scripts
195+
./start-server.sh
196+
```
197+
198+
### 7. Verify end-to-end
199+
200+
Push a commit or open a PR/issue on your repository. You should see:
201+
- Log output in the server console
202+
- Events in the SSE stream (`./scripts/stream-events.sh`)
203+
- Live updates in `viewer/index.html`
204+
205+
### Troubleshooting Dev Tunnels
206+
207+
| Issue | Solution |
208+
|-------|----------|
209+
| `devtunnel: command not found` | Ensure the CLI is installed and on your PATH |
210+
| GitHub shows "delivery failed" | Verify the tunnel is running and the port matches `WEBHOOK_PORT` in `.env` |
211+
| 401 from Drasi | Ensure `GITHUB_WEBHOOK_SECRET` in `.env` matches the secret in GitHub webhook settings |
212+
| Tunnel URL changed | Dev tunnels get a new URL each time unless you use a named tunnel (see persistent tunnel tip above) |
213+
214+
### Alternative: Using ngrok
215+
216+
If you prefer [ngrok](https://ngrok.com/):
217+
218+
```bash
219+
ngrok http 9000
220+
```
221+
222+
Then use the ngrok URL (e.g., `https://abc123.ngrok-free.app/github/events`) as the Payload URL in GitHub.
223+
224+
## Configuration Deep Dive
225+
226+
### Webhook Source Configuration
227+
228+
The HTTP source uses **webhook mode** (enabled by the `webhooks` section) which provides:
229+
230+
- **Route matching**: Path patterns with optional path parameters
231+
- **Authentication**: HMAC-SHA256 signature verification matching GitHub's format
232+
- **Conditional mappings**: Route different event types based on headers
233+
- **Template engine**: Handlebars templates to extract fields from payloads
234+
235+
### Key Configuration Patterns
236+
237+
**Routing by event type** using the `when` condition:
238+
```yaml
239+
- when:
240+
header: X-GitHub-Event
241+
equals: push
242+
```
243+
244+
**Extracting nested fields** using Handlebars dot notation:
245+
```yaml
246+
template:
247+
id: "push-{{payload.head_commit.id}}"
248+
properties:
249+
author: "{{payload.head_commit.author.username}}"
250+
```
251+
252+
**HMAC signature verification** matching GitHub's format:
253+
```yaml
254+
auth:
255+
signature:
256+
type: hmac-sha256
257+
secretEnv: GITHUB_WEBHOOK_SECRET
258+
header: X-Hub-Signature-256
259+
prefix: "sha256="
260+
encoding: hex
261+
```
262+
263+
### Queries
264+
265+
| Query | Description |
266+
|-------|-------------|
267+
| `recent-pushes` | Push events with commit details |
268+
| `open-pull-requests` | Pull requests in "open" state |
269+
| `issues-opened` | Newly opened issues |
270+
271+
## Extending This Example
272+
273+
### Add more event types
274+
275+
GitHub supports many event types. Add new mappings for events like:
276+
- `star` — Track repository stars
277+
- `workflow_run` — Track CI/CD activity
278+
- `release` — Track new releases
279+
280+
Example mapping for star events:
281+
```yaml
282+
- when:
283+
header: X-GitHub-Event
284+
equals: star
285+
operation: insert
286+
elementType: node
287+
template:
288+
id: "star-{{payload.sender.id}}"
289+
labels: [StarEvent]
290+
properties:
291+
repo: "{{payload.repository.full_name}}"
292+
action: "{{payload.action}}"
293+
user: "{{payload.sender.login}}"
294+
```
295+
296+
### Add an HTTP reaction
297+
298+
Forward processed events to another service (e.g., Slack, Discord):
299+
300+
```yaml
301+
reactions:
302+
- kind: http
303+
id: slack-notifications
304+
queries:
305+
- open-pull-requests
306+
autoStart: true
307+
baseUrl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
308+
routes:
309+
open-pull-requests:
310+
added:
311+
url: ""
312+
method: POST
313+
body: '{"text": "New PR #{{after.PRNumber}}: {{after.Title}} by {{after.Author}}"}'
314+
```
315+
316+
## Troubleshooting
317+
318+
### Webhook returns 401 Unauthorized
319+
320+
The HMAC signature doesn't match. Ensure:
321+
- `GITHUB_WEBHOOK_SECRET` in `.env` matches the secret configured in GitHub
322+
- The simulation scripts are using the same secret
323+
324+
### Events aren't appearing in queries
325+
326+
Check the server logs for mapping errors. Common issues:
327+
- Payload field paths don't match (e.g., GitHub changed their webhook schema)
328+
- The `X-GitHub-Event` header value doesn't match the `when.equals` condition
329+
330+
### SSE connection drops
331+
332+
The SSE reaction has a 30-second heartbeat. If your browser disconnects, it should reconnect automatically (the viewer page handles this).
333+
334+
## Related Examples
335+
336+
- [HTTP Webhook Receiver](../configs/02-sources/http-webhook-receiver.yaml) — Simpler HTTP source without webhook mode
337+
- [HTTP Webhook Sender](../configs/03-reactions/http-webhook-sender.yaml) — HTTP reaction for outbound webhooks
338+
- [Getting Started](../getting-started/) — PostgreSQL CDC tutorial

0 commit comments

Comments
 (0)