Skip to content

runbook.md

Latest commit

 

History

History
154 lines (100 loc) · 3.97 KB

runbook.md

File metadata and controls

154 lines (100 loc) · 3.97 KB
Raw

Runbook

Service model

  • One stream process per source instance.
  • One periodic maintain process (oneshot) driven by a timer.

Baseline templates are in examples/systemd/.

Deployment workflow

1) Prepare host and directories

  1. Ensure Python 3.10+ and mysql client tools (mysql, mysqlbinlog) are available.
  2. Install runtime (venv or system-wide) so mysql-binlog-archiver is executable.
  3. Create service user and runtime directories.

Example:

sudo useradd --system --home /var/lib/mysqlbinlog_archiver --shell /usr/sbin/nologin archiver
sudo install -d -o archiver -g archiver -m 0750 /var/lib/mysqlbinlog_archiver/state
sudo install -d -o archiver -g archiver -m 0750 /var/lib/mysqlbinlog_archiver/reports
sudo install -d -o archiver -g archiver -m 0750 /var/log/mysqlbinlog_archiver
sudo install -d -o root -g root -m 0755 /etc/mysqlbinlog_archiver/instances
sudo install -d -o root -g root -m 0755 /etc/mysqlbinlog_archiver/credentials

2) Install instance and credentials config

  1. Create one JSON file per source in /etc/mysqlbinlog_archiver/instances/.
  2. Create one defaults-extra-file per source in /etc/mysqlbinlog_archiver/credentials/.
  3. Set credentials files to mode 0600.

3) Install and adjust systemd units

  1. Copy templates from examples/systemd/ into /etc/systemd/system/.
  2. Verify ExecStart points to the correct binary path (command -v mysql-binlog-archiver).
  3. Keep instance name aligned with %i.json convention.
sudo cp examples/systemd/mysqlbinlog-archiver@.service /etc/systemd/system/
sudo cp examples/systemd/mysqlbinlog-archiver-maintain.service /etc/systemd/system/
sudo cp examples/systemd/mysqlbinlog-archiver-maintain.timer /etc/systemd/system/
sudo systemctl daemon-reload

4) Enable and start services

sudo systemctl enable --now mysqlbinlog-archiver@mysql-01.example.com.service
sudo systemctl enable --now mysqlbinlog-archiver-maintain.timer

Common checks

  1. Check stream service state.
sudo systemctl status mysqlbinlog-archiver@mysql-01.example.com.service
  1. Check maintain timer and last run.
sudo systemctl status mysqlbinlog-archiver-maintain.timer
sudo systemctl status mysqlbinlog-archiver-maintain.service
  1. Inspect logs.
sudo journalctl -u mysqlbinlog-archiver@mysql-01.example.com.service -n 100 --no-pager
sudo journalctl -u mysqlbinlog-archiver-maintain.service -n 100 --no-pager
  1. Inspect state and report artifacts.
  • /var/lib/mysqlbinlog_archiver/state/<source_id>.json
  • /var/lib/mysqlbinlog_archiver/state/_maintain.json
  • /var/lib/mysqlbinlog_archiver/reports/maintain-latest.json
  1. Confirm binlog files appear and grow in the configured binlog_dir.

Troubleshooting

Connectivity failures

Symptoms:

  • status=connectivity or last_error_type=connectivity_bootstrap
  • stream retries with backoff

Actions:

  • Verify source host DNS resolution.
  • Verify TCP reachability to source port.
  • Verify firewall/network policy.

Authentication failures

Symptoms:

  • last_error_type=auth

Actions:

  • Verify credentials in defaults-extra-file.
  • Verify source grants for binlog streaming user.

Source gap / purged binlog

Symptoms:

  • last_error_type=source_gap

Actions:

  • Check source retention and purge policy.
  • Adjust bootstrap policy/start file as needed.

Disk pressure

Symptoms:

  • status=disk_full

Actions:

  • Free disk/inodes.
  • Confirm retention thresholds are reasonable.
  • Verify archiver user can delete old closed files.

Recovery and rollback basics

  1. Restart one stream unit after config fix:
sudo systemctl restart mysqlbinlog-archiver@mysql-01.example.com.service
  1. Temporarily stop maintain timer:
sudo systemctl disable --now mysqlbinlog-archiver-maintain.timer
  1. Re-enable timer after remediation:
sudo systemctl enable --now mysqlbinlog-archiver-maintain.timer

On successful stream restart, state should return to running and clear previous error fields.