This guide is for wiring a board into whatever your organisation already uses to watch its infrastructure: what to scrape, what to alert on, and how the pieces relate to each other. It assumes the Compose deployment — web, worker, postgres, and migrate running to completion.
None of this is required to run a board. Every feature here is off unless you turn it on, and a board with nothing configured still logs, still answers /api/health, and still drives its own container healthchecks.
Two endpoints answer different questions, both unauthenticated (neither returns anything an anonymous visitor could not already infer from the board being up):
-
/api/health— can this process serve a request at all. Answers{ok:true}unconditionally once the server has bound its port. It says nothing about the database on purpose: an orchestrator that killed every web replica the moment Postgres blinked would turn a two-second blip into a two-minute outage while replicas restart and reconnect. -
/api/ready— can this board do its job. Reuses the same health data the admin panel's System Health page is built from (assessScheduler): it checks that the database answers, and that the scheduler is notschedulerStopped— every enabled task gone stale at once, the signal that the worker itself has died rather than one task having a bad run. Answers200when both hold,503otherwise, with the detail in the body:{ "ok": true, "at": "2026-08-21T12:00:00.000Z", "database": { "ok": true }, "scheduler": { "ok": true, "schedulerStopped": false, "stale": 0, "failing": 0 } }staleandfailingare reported for visibility but do not by themselves flipoktofalse— a single misbehaving task should show up in the admin panel and its own alert (below), not restart every container in the fleet.
docker/healthcheck.sh — the image's own HEALTHCHECK, driving docker compose ps and depends_on: condition: service_healthy — calls /api/ready for the web role. This board runs a single, non-clustered Postgres, so the fleet-wide-blip risk /api/health is deliberately blind to does not apply to the container healthcheck the way it would behind a database with automatic failover. The worker role has no HTTP server to probe, so its healthcheck runs the same check as a one-shot process instead: node apps/worker/worker.cjs --ready, which reuses the identical database-and-scheduler check.
The Compose deployment runs apps/worker, a long-lived process that ticks every 60 seconds. Where no long-lived process exists — a serverless deployment, or a board whose host only offers a cron scheduler — the same tick is available over HTTP at /api/system/tick, and an external scheduler calls it on a schedule instead. Nothing else changes: the HTTP tick runs the identical tick() over the identical task list, and tasks claim their work through the database (see Architecture), so an overlapping call from a retry, a second instance, or a worker still running alongside it cannot double-process anything.
GET and POST both run one tick, so a scheduler that only issues one of them works either way.
The endpoint accepts a shared secret in two variables, and either one on its own protects it:
| Variable | Presented as | For |
|---|---|---|
TICK_SECRET |
Authorization: Bearer <TICK_SECRET>, or an X-Tick-Secret header |
Any caller: curl, a systemd timer, a GitHub Actions workflow, an uptime pinger |
CRON_SECRET |
Authorization: Bearer <CRON_SECRET> |
Vercel Cron, which sends exactly this header under exactly this variable name and cannot be told to send another |
Both are compared in constant time, and a caller presenting the wrong one — or nothing — gets a plain 404, the same answer /api/metrics gives an unauthenticated scraper: the endpoint does not admit it exists.
Both are held to the same floor as every other secret this board reads, 32 characters, and the board refuses to boot on a shorter one. Note that this is stricter than the 16 characters Vercel's own cron documentation suggests for CRON_SECRET: a value generated by following those instructions is rejected here, and the fix is a longer secret rather than a lower floor. Generate one with:
openssl rand -hex 32Set whichever the scheduler in front of the board can actually send. Setting both is fine and is how a board moves from one to the other without downtime — during the overlap either is accepted. The secret is never read from the query string; a caller that puts it there is refused and told so in the log.
With neither set, the tick runs unauthenticated and logs a warning on every call. That is a development affordance and nothing else: in production the board refuses to boot unless at least one of the two is set, the same way it refuses to boot without AUTH_SECRET.
A Vercel deployment declares its schedule in vercel.json at the root of the project it deploys:
{
"crons": [{ "path": "/api/system/tick", "schedule": "* * * * *" }]
}Vercel calls that path with Authorization: Bearer <CRON_SECRET> on the schedule given, and the every-minute schedule above is what matches the worker's own loop.
Hobby projects cannot have that schedule. The plan allows a couple of cron jobs and runs each of them about once a day, at an hour Vercel chooses; only paid plans accept an arbitrary cron expression. A board driven by Hobby cron therefore ticks daily, which — by the latency note under Cadence — makes "as it happens" notifications a daily digest in all but name. A board that wants a minute-by-minute tick on Hobby drives the endpoint from something else that can call a URL on a schedule: a GitHub Actions workflow, a systemd timer, an uptime pinger, using TICK_SECRET.
{ "ok": true, "ran": [ { "taskId": "outbox.relay", "status": "ran", "durationMs": 12 } ], "registered": 21 }| Status | Meaning | What a scheduler should do |
|---|---|---|
200, ok: true |
Every task the tick reached ran or was skipped because it was not due | Nothing |
200, ok: false |
The tick itself completed; at least one task in ran has status: "failed" with its error |
Nothing automatic — look at the failing taskId |
404 |
The secret was wrong or absent | Fix the secret; retrying will not help |
503 |
This board has no scheduler, because DATA_SOURCE=fixture. A tick needs durable, cross-instance state to guarantee a task is not run twice |
Nothing; a fixture board has no work to do |
A failed task is deliberately not an error status. A tick that reached the tasks and ran them did its job, even when one of those tasks threw — so the response is 200 with ok: false rather than a 5xx. The reason is the retry behaviour on the other side: schedulers retry non-2xx responses, and a task that fails every time it runs would turn each retry into another attempt, tight-looping a poisoned task and hammering whatever it fails against. A failing task instead surfaces where failures are meant to surface — the system.task_failed administrator notification, meith_task_runs_total{status="error"}, the admin panel's System Health page, and /api/ready's scheduler.failing — while the scheduler simply calls again on its next scheduled tick. Only a genuinely unusable endpoint (404, 503) answers outside the 2xx range.
Every task carries its own intervalSeconds and is skipped when it is not due, so calling the endpoint more often than the shortest interval costs a claim query per task and nothing else. Calling it less often than 60 seconds does not break anything either — tasks are written so that a missed run delays work rather than losing it — but it does stretch the latency of everything the shortest-interval tasks drive: relaying domain events onto the queue, draining that queue, and the "as it happens" subscription notifications, all of which want 60 seconds. At an hourly cadence a reply notification can be an hour late; at a daily one, a day. Nothing is lost, but "instant" stops meaning instant.
The route declares maxDuration = 300, so a host that reads Next.js's build output (Vercel does) allows a long tick rather than cutting it off at a default. That ceiling is chosen to cover the worst case: the per-minute tasks' own budgets — webhook delivery's 240 seconds being much the largest — can add up to roughly six minutes if every one of them runs to its limit in the same tick, and each task is aborted at its own budget regardless. A tick killed mid-run leaves its claims to expire after 15 minutes, and the next tick picks the work up.
On Vercel that figure is checked when the project builds, not when the function runs, so a plan that does not allow it fails the deployment rather than clamping the request. With Fluid Compute — the default for new projects — Hobby allows up to 300 seconds and the declaration builds as written. With Fluid Compute switched off, Hobby caps a function at 60 seconds and this build fails. Turn Fluid Compute back on, or lower the constant in apps/community/app/api/system/tick/route.ts to what the plan allows; a lower ceiling costs nothing but the tail of an unusually long tick, which the next tick picks up anyway.
A bounded task trades reach per run for a fixed cost per run, and that is a trade rather than a speed-up. promotions.apply is the clearest case. It used to evaluate every member on the board every six hours, so its cost grew with the membership and a large board risked a run that outlived the function it ran in. It now evaluates at most 10,000 members an hour — 240,000 a day — resuming from a stored cursor, so one run costs the same whatever the board's size. Below roughly 240,000 members the new scheme comes round at least as often as the old one did. Above it, it is slower: a board of a million members went from a full pass every six hours to a full pass every four days or so. Bounding is still the right call, because a promotion arriving late is a delay while a run killed mid-scan is work that never lands — but it is worth knowing which side of that line a board sits on. The page size and the interval are constants in packages/tasks/src/builtin.ts; there is no setting for them.
Off by default. Turn it on with:
METRICS_ENABLED=1
METRICS_TOKEN=<generated, openssl rand -base64 32>METRICS_TOKEN is required in production once METRICS_ENABLED is set — the board refuses to boot without it, the same way it refuses to boot without AUTH_SECRET. Outside production an unset token serves the endpoint unauthenticated and logs a warning, for a local Prometheus without the ceremony.
/api/metrics answers in the Prometheus text exposition format, authenticated the same way as /api/system/tick: present the token as Authorization: Bearer <METRICS_TOKEN>, or as an X-Metrics-Token header for a scraper that cannot set Authorization. A request with METRICS_ENABLED off, or the wrong token, gets a plain 404 — the endpoint does not admit it exists to an unauthenticated caller.
# prometheus.yml
scrape_configs:
- job_name: meith-web
metrics_path: /api/metrics
authorization:
credentials: <METRICS_TOKEN>
static_configs:
- targets: ['board.example:443']| Metric | Type | Labels | What it means |
|---|---|---|---|
meith_task_runs_total |
counter | task, status (ok|error) |
Scheduled task runs. A rising error rate for one task is the first place to look before meith_task_run_duration_seconds. |
meith_task_run_duration_seconds |
histogram | task |
How long each scheduled task takes. Compare against docs/reference/performance.md's offline budgets when a task starts running long. |
meith_http_request_duration_seconds |
histogram | method, route, status |
REST API v1 (/api/v1/*) request duration. route is the declared path template (/threads/:id), not the URL, so cardinality stays bounded regardless of how many distinct threads or forums are requested. |
meith_queue_jobs |
gauge | status (queued|dead) |
Jobs pending or running, and jobs in the dead-letter state, read fresh from Postgres at scrape time. |
meith_db_connections_active |
gauge | — | Server-side connections to this database (pg_stat_activity), read fresh at scrape time. |
Counters and histograms accumulate for the life of the process; the two gauges are refreshed from the database on every scrape rather than kept continuously in sync. Each web instance and the worker is its own scrape target with its own numbers — see Scaling out for what that means once there is more than one.
meith_queue_jobs{status="dead"}rising — jobs are failing every retry and landing in the dead-letter state. Not urgent by itself (nothing is silently dropped), but a queue nobody looks at is where a broken integration hides.meith_queue_jobs{status="queued"}growing without bound — the worker is falling behind or has stopped. Cross-check against/api/ready'sscheduler.stale.up{job="meith-web"} == 0or repeated/api/readyfailures — the standard "is it up" alert, sourced from Prometheus's own scrape health or a separate uptime check against/api/ready.- A step change in
meith_task_run_duration_secondsfor a specifictask— the offlinepnpm perfharness catches regressions before release; this is the same signal in production, after release. meith_http_request_duration_secondsp95 rising for the REST API — the question this ticket started from ("is p95 thread render getting worse") now has an answer that does not require re-running the offline harness.
None of these are wired to a specific alerting tool by design — Prometheus's own Alertmanager, Grafana alerting, or a hosted equivalent all read the same metric names.
Off by default, and, unlike metrics, has no companion secret to configure — a trace exporter either has a collector to talk to or it does not.
OTEL_ENABLED=1
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4318/v1/tracesWhen enabled, the web server (in instrumentation.ts) and the worker (at the start of its main loop) each register an OpenTelemetry NodeTracerProvider exporting over OTLP/HTTP in batches. Two spans exist today: task.run around every scheduled task, and http.request around every REST API v1 request — both carry the same labels their metrics counterparts use (task.id; http.method, http.route, http.status_code), so a slow histogram bucket and a slow trace point at the same thing.
With OTEL_ENABLED unset, every call into the tracing API is a documented no-op — the OpenTelemetry SDK is imported dynamically and only constructed once the flag is on, so a board that never turns this on never pays for it beyond the dependency's presence in the image.
Nothing here replaces docs/reference/performance.md's offline harness — tracing shows where time went for one request in production; the harness is still how you catch a regression before it ships.
Structured JSON logs (pino, request-ID correlated, secrets redacted — see packages/core/src/logger.ts) go to stdout already; nothing here changes that. The Compose file caps each container's own json-file log driver at 10MB × 3 files, which is retention on the machine, not shipping.
To get logs into a log aggregator, point a collector at the container runtime rather than the application: Vector, Promtail or Fluent Bit can all tail Docker's own JSON log files or read directly from the Docker/containerd log driver, and every line is already a JSON object — no parsing rules to write. docker compose logs --since 1h web worker, from Operations, remains the right tool for looking at recent logs by hand.
| Question | Where |
|---|---|
| Is the process alive? | /api/health |
| Can the board do its job? | /api/ready, and the container healthchecks it drives |
| Can the board run scheduled work without a worker process? | /api/system/tick, called by a cron scheduler |
| What are current numbers? | /api/metrics (METRICS_ENABLED=1) |
| Where did this request's time go? | Tracing (OTEL_ENABLED=1) |
| What happened, in order? | Structured logs, shipped from the container runtime |
| Rich detail on tasks, search index, recounts, mail | The admin panel's System Health page — see the Organiser guide |