[need help] validation error: cannot wait for replica: replica wait exceeded timeout

[roelvan]

Hi guys!

Can anyone point me in the right direction with this error:

validation error: cannot wait for replica: replica wait exceeded timeout

My litestream backups used to work well till a few days ago. The error shows when I run sudo journalctl -u litestream -f.

My config (aws credentials are working):

access-key-id: SECRET
secret-access-key: SECRET

dbs:
  - path: /var/www/DB/uh.db
    replicas:
      - url: s3://...
        sync-interval: 60s
        validation-interval: 24h

Using v0.3.7

Thanks!

[benbjohnson]

hey @roelvan. Sorry for the delay getting back to you. Validation has a timing bug with how it obtains locks on the database so it can have false positives. It's being removed in v0.4.0 and you should manually call litestream restore to validate backups instead.

[roelvan]

The problem is that litestream is not backing up anything at all. When I try to restore I get a version from a few weeks ago.

[benbjohnson]

@roelvan When you make changes to your database, do you see new .wal.lz4 files uploaded to S3? It sounds like Litestream either isn't running or isn't monitoring the right database path.

[tiguere]

i'm having the same issue. i am seeing .wal.lz4 files being created, but then, later on, the restored db is outdated...

configuration file looks like:

exec: gunicorn --config gunicorn-cfg.py config.wsgi
dbs:
  - path: ./data/db.sqlite3
    replicas:
      - type:   gcs
        bucket: durodev-prod-data
        path:   backups
        validation-interval: 8h

on startup, the following bash script is executed:

#!/usr/bin/env bash

source ./.env

if [[ -z "$DB_DIR" ]]; then
    echo "DB_DIR env var not specified"
    exit 1
fi
if [[ -z "$REPLICA_URL" ]]; then
    echo "REPLICA_URL env var not specified"
    exit 1
fi

mkdir -p "{$DB_DIR}"

litestream restore -if-db-not-exists -if-replica-exists -o "./$DB_DIR/db.sqlite3" "$REPLICA_URL"

./manage.py migrate --noinput
./manage.py createcachetable

chmod -R a+rwX "./$DB_DIR"
exec litestream replicate -config litestream.yml

[roelvan]

No worries! Thanks for the clarification. Will check it out.