Skip to content

Add heartbeat to background worker epoch sentinel - #42

Merged
Pino de Candia (pinodeca) merged 2 commits into
mainfrom
heartbeat-sentinel
Mar 6, 2026
Merged

Add heartbeat to background worker epoch sentinel#42
Pino de Candia (pinodeca) merged 2 commits into
mainfrom
heartbeat-sentinel

Conversation

@pinodeca

@pinodeca Pino de Candia (pinodeca) commented Mar 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a last_seen_at column to df._worker_epoch and update the epoch sentinel check to write a heartbeat on every poll tick (~5 seconds). This gives operators a simple SQL query to verify background worker liveness without inspecting logs or process lists.

Changes

  • src/lib.rs: Add last_seen_at TIMESTAMPTZ column to df._worker_epoch DDL
  • src/worker.rs: Update sentinel insert to include last_seen_at; replace SELECT EXISTS(...) with UPDATE ... SET last_seen_at = now() ... RETURNING epoch_id
  • USER_GUIDE.md: Add Worker Liveness subsection to Monitoring
  • docs/extension_lifecycle.md: Update schema description to include new column
  • tests/e2e/sql/35_heartbeat_liveness.sql: E2E test that verifies last_seen_at advances over time
  • scripts/test-e2e-local.sh: Register new test as superuser (reads internal df._worker_epoch)

User-Facing Liveness Check

SELECT epoch_id, started_at, last_seen_at,
       now() - last_seen_at AS time_since_last_heartbeat
  FROM df._worker_epoch;
  • time_since_last_heartbeat < 15 seconds → worker is alive (recent heartbeat)
  • No rows or large gap → worker is likely down or hasn't initialized yet

Cost

  • No new queries — the existing epoch-check SELECT is replaced with an UPDATE ... RETURNING (same round trip)
  • One additional column written per poll tick (~5s) — negligible I/O

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a heartbeat timestamp to the background worker epoch sentinel so operators can query worker liveness via SQL, without relying on logs or process inspection.

Changes:

  • Extend df._worker_epoch with a last_seen_at column.
  • Make the epoch-sentinel poll update last_seen_at (heartbeat) on each tick via UPDATE ... RETURNING.
  • Document the new liveness query in the user guide and lifecycle docs.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/worker.rs Writes last_seen_at on sentinel creation and updates it on each poll tick.
src/lib.rs Adds last_seen_at TIMESTAMPTZ to df._worker_epoch table DDL.
USER_GUIDE.md Adds a “Worker Liveness” section with a query and interpretation guidance.
docs/extension_lifecycle.md Updates epoch-sentinel schema description to include last_seen_at.

Comment thread src/worker.rs
Comment thread USER_GUIDE.md Outdated
Comment thread src/lib.rs
Add last_seen_at column to df._worker_epoch and update the epoch
sentinel check from a SELECT to UPDATE ... RETURNING, writing
now() on every poll tick (~5s). This gives operators a simple
SQL query to verify background worker liveness.

Update USER_GUIDE.md with Worker Liveness monitoring section.
- Consolidate USER_GUIDE.md liveness bullets and drop 'healthy' claim
- Add E2E test for heartbeat liveness (35_heartbeat_liveness.sql)
@pinodeca
Pino de Candia (pinodeca) merged commit 40cd3ff into main Mar 6, 2026
5 checks passed
@pinodeca
Pino de Candia (pinodeca) deleted the heartbeat-sentinel branch March 6, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants