Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/http-postgres.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
branches: [main]
paths:
- 'http.dockerfile'
- 'pgsql-http.dockerfile'
- '.github/workflows/http-postgres.yml'
workflow_dispatch:

Expand All @@ -22,4 +22,4 @@ jobs:
build_target: pg-${{ matrix.pg_version }}
image_tags: |
"ghcr.io/hotosm/postgres:${{ matrix.pg_version }}-http"
dockerfile: http.dockerfile
dockerfile: pgsql-http.dockerfile
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
runs-on: ubuntu-latest

container:
image: docker.io/goreleaser/goreleaser-cross:v1.24
image: docker.io/goreleaser/goreleaser-cross:v1.25

steps:
- name: Checkout
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM golang:1.24 AS base
FROM golang:1.25 AS base


# Build statically compiled binary
Expand Down
169 changes: 56 additions & 113 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<!-- markdownlint-disable -->
<p align="center">
<em>A lightweight webhook for ODK Central submissions and entity property updates.</em>
<em>A lightweight CLI tool for installing webhook triggers in ODK Central database.</em>
</p>
<p align="center">
<a href="https://github.com/hotosm/central-webhook/actions/workflows/release.yml" target="_blank">
Expand All @@ -17,75 +17,63 @@

<!-- markdownlint-enable -->

Call a remote API on ODK Central database events:
Install PostgreSQL triggers that automatically call remote APIs on ODK Central database events:

- New submission (XML).
- Update entity (entity properties).
- Submission review (approved, hasIssues, rejected).

The `centralwebhook` binary is small ~15MB and only consumes
~5MB of memory when running.

> [!NOTE]
> There is a 8000 byte Postgres limit for the submission XML that can be sent
> until [this issue](https://github.com/hotosm/central-webhook/issues/9)
> is addressed.
>
> The submission XML will simply be truncated in this case.
The `centralwebhook` tool is a simple CLI that installs or uninstalls database triggers. Once installed, the triggers use the `pgsql-http` extension to send HTTP requests directly from the database.

## Prerequisites

- ODK Central running, connecting to an accessible Postgresql database.
- A POST webhook endpoint on your service API, to call when the selected
event occurs.
- ODK Central running, connecting to an accessible PostgreSQL database.
- The `pgsql-http` extension installed and enabled in your PostgreSQL database:
```sql
CREATE EXTENSION http;
```
> [!NOTE]
> **Using our helper images**: We provide PostgreSQL images with the `pgsql-http` extension pre-installed:
> - `ghcr.io/hotosm/postgres:18-http` (based on vanilla PostgreSQL 18 images)
>
> These images are drop-in replacements for standard PostgreSQL images and simply add the extension.
>
> **Installing manually**: If you don't wish to use these images, you must install the `pgsql-http` extension yourself. The extension may require superuser privileges to install. If you cannot install it yourself, ask your database administrator.
- A POST webhook endpoint on your service API, to call when the selected event occurs.

## Usage

The `centralwebhook` tool is a service that runs continually, monitoring the
ODK Central database for updates and triggering the webhook as appropriate.
The `centralwebhook` tool is a CLI that installs or uninstalls database triggers. After installation, the triggers run automatically whenever audit events occur in the database.

### Integrate Into [ODK Central](https://github.com/getodk/central) Stack
### Install Triggers

- It's possible to include this as part of the standard ODK Central docker
compose stack.
- First add the environment variables to your `.env` file:
Install webhook triggers in your database:

```dotenv
CENTRAL_WEBHOOK_UPDATE_ENTITY_URL=https://your.domain.com/some/webhook
CENTRAL_WEBHOOK_REVIEW_SUBMISSION_URL=https://your.domain.com/some/webhook
CENTRAL_WEBHOOK_NEW_SUBMISSION_URL=https://your.domain.com/some/webhook
CENTRAL_WEBHOOK_API_KEY=your_api_key_key
```
```bash
./centralwebhook install \
-db 'postgresql://{user}:{password}@{hostname}/{db}?sslmode=disable' \
-updateEntityUrl 'https://your.domain.com/some/webhook' \
-newSubmissionUrl 'https://your.domain.com/some/webhook' \
-reviewSubmissionUrl 'https://your.domain.com/some/webhook'
```

> [!TIP]
> Omit a xxx_URL variable if you do not wish to use that particular webhook.
> Omit a webhook URL flag if you do not wish to use that particular webhook.
>
> The CENTRAL_WEBHOOK_API_KEY variable is also optional, see the
> [APIs With Authentication](#apis-with-authentication) section.

- Then extend the docker compose configuration at startup:

```bash
# Starting from the getodk/central code repo
docker compose -f docker-compose.yml -f /path/to/this/repo/compose.webhook.yml up -d
```

### Other Ways To Run
> The `-apiKey` flag is optional, see the [APIs With Authentication](#apis-with-authentication) section.

<details>
<summary>Via Docker (Standalone)</summary>
### Uninstall Triggers

#### Via Docker (Standalone)
Remove webhook triggers from your database:

```bash
docker run -d ghcr.io/hotosm/central-webhook:latest \
-db 'postgresql://{user}:{password}@{hostname}/{db}?sslmode=disable' \
-updateEntityUrl 'https://your.domain.com/some/webhook' \
-newSubmissionUrl 'https://your.domain.com/some/webhook' \
-reviewSubmissionUrl 'https://your.domain.com/some/webhook'
./centralwebhook uninstall \
-db 'postgresql://{user}:{password}@{hostname}/{db}?sslmode=disable'
```

Environment variables are also supported:
### Environment Variables

All flags can also be provided via environment variables:

```dotenv
CENTRAL_WEBHOOK_DB_URI=postgresql://user:pass@localhost:5432/db_name?sslmode=disable
Expand All @@ -96,74 +84,22 @@ CENTRAL_WEBHOOK_API_KEY=ksdhfiushfiosehf98e3hrih39r8hy439rh389r3hy983y
CENTRAL_WEBHOOK_LOG_LEVEL=DEBUG
```

</details>

<details>
<summary>Via Binary (Standalone)</summary>
### Via Docker

#### Via Binary (Standalone)

Download the binary for your platform from the
[releases](https://github.com/hotosm/central-webhook/releases) page.

Then run with:
You can run the CLI tool via Docker:

```bash
./centralwebhook \
docker run --rm ghcr.io/hotosm/central-webhook:latest install \
-db 'postgresql://{user}:{password}@{hostname}/{db}?sslmode=disable' \
-updateEntityUrl 'https://your.domain.com/some/webhook' \
-newSubmissionUrl 'https://your.domain.com/some/webhook' \
-reviewSubmissionUrl 'https://your.domain.com/some/webhook'
```

> It's possible to specify a single webhook event, or multiple.

</details>

<details>
<summary>Via Code</summary>

#### Via Code

Usage via the code / API:

```go
package main

import (
"fmt"
"context"
"log/slog"

"github.com/hotosm/central-webhook/db"
"github.com/hotosm/central-webhook/webhook"
)

ctx := context.Background()
log := slog.New()

dbPool, err := db.InitPool(ctx, log, "postgresql://{user}:{password}@{hostname}/{db}?sslmode=disable")
if err != nil {
fmt.Fprintf(os.Stderr, "could not connect to database: %v", err)
}

err = SetupWebhook(
log,
ctx,
dbPool,
nil,
"https://your.domain.com/some/entity/webhook",
"https://your.domain.com/some/submission/webhook",
"https://your.domain.com/some/review/webhook",
)
if err != nil {
fmt.Fprintf(os.Stderr, "error setting up webhook: %v", err)
}
```

> To not provide a webhook for an event, pass `nil` as the url.
### Download Binary

</details>
Download the binary for your platform from the
[releases](https://github.com/hotosm/central-webhook/releases) page.

## Webhook Request Payload Examples

Expand Down Expand Up @@ -208,7 +144,7 @@ Many APIs will not be public and require some sort of authentication.
There is an optional `-apiKey` flag that can be used to pass
an API key / token provided by the application.

This will be inserted in the `X-API-Key` request header.
This will be inserted in the `X-API-Key` request header when the trigger sends HTTP requests.

No other authentication methods are supported for now, but feel
free to open an issue (or PR!) for a proposal to support other
Expand All @@ -217,7 +153,7 @@ auth methods.
Example:

```bash
./centralwebhook \
./centralwebhook install \
-db 'postgresql://{user}:{password}@{hostname}/{db}?sslmode=disable' \
-updateEntityUrl 'https://your.domain.com/some/webhook' \
-apiKey 'ksdhfiushfiosehf98e3hrih39r8hy439rh389r3hy983y'
Expand Down Expand Up @@ -284,18 +220,25 @@ async def update_entity_status_in_fmtm(
raise HTTPException(status_code=400, detail=msg)
```

## How It Works

The tool installs PostgreSQL triggers on the `audits` table that:

1. Detect when audit events occur (entity updates, submission creates/updates)
2. Format the event data into a JSON payload with `type`, `id`, and `data` fields
3. Use the `pgsql-http` extension to send an HTTP POST request directly from the database
4. Include the `X-API-Key` header if provided during installation

The triggers run automatically after installation - no long-running service is needed.

## Development

- This package mostly uses the standard library, plus a Postgres driver
and testing framework.
- This package uses the standard library and a Postgres driver.
- Binary and container image distribution is automated on new **release**.

### Run The Tests

The test suite depends on a database, so the most convenient way is to run
via docker.

There is a pre-configured `compose.yml` for testing:
The test suite depends on a database with the `pgsql-http` extension installed. The most convenient way is to run via docker:

```bash
docker compose run --rm webhook
Expand Down
3 changes: 3 additions & 0 deletions compose.webhook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ services:
postgres14:
condition: service_started
restart: always

postgres14:
image: "ghcr.io/hotosm/postgres:14-http"
7 changes: 4 additions & 3 deletions compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ services:
- ./main.go:/app/main.go:ro
- ./main_test.go:/app/main_test.go:ro
- ./db:/app/db:ro
- ./webhook:/app/webhook:ro
- ./parser:/app/parser:ro
# environment:
# # Override to use database on host
# CENTRAL_WEBHOOK_DB_URI: postgresql://odk:odk@host.docker.internal:5434/odk?sslmode=disable
Expand All @@ -36,7 +34,10 @@ services:
entrypoint: go test -timeout=2m -v ./...

db:
image: "postgis/postgis:17-3.5-alpine"
image: "ghcr.io/hotosm/postgres:18-http"
build:
dockerfile: pgsql-http.dockerfile
target: pg-18
container_name: centralwebhook-db
environment:
- POSTGRES_USER=odk
Expand Down
Loading