Litestream as a secondary backup for Trilium

[kalbasit]

Hello, thanks to all the contributors that made this possible; Recent user of Trilium moving/moved from Obsidian.

I have my Trilium deployed on my Kubernetes cluster (btw the Helm chart is out of date, using the old notes application). I have placed Litestream in a native sidecar to continuously stream SQLite WAL to MinIO as a secondary backup (located in a different location for me).

The setup works fine for some time, but there are times when Trilium reports The database is locked. and when that happens it gets stuck until I start Trilium without the Litestream sidecar. I tried to set SQLITE_BUSY_TIMEOUT=30000 hoping it may help (as recommended on Litestream documentation) but it did not help. I also tried to force HTTPv1 on Litestream (as I saw few stream errors relating to HTTPv2, but did not make a difference).

Has anyone tried to use Trilium with Litestream?

Log with error message

ERROR: put /api/notes/:noteId/data threw exception: 'database is locked', stack: SqliteError: database is locked
    at /usr/src/app/main.cjs:963:1214
    at Q5 (/usr/src/app/main.cjs:963:1591)
    at Ofe (/usr/src/app/main.cjs:963:1204)
    at Object.b6i [as upsert] (/usr/src/app/main.cjs:963:115)
    at LQe.saveBlob (/usr/src/app/main.cjs:27:4288)
    at sqliteTransaction.<anonymous> (/usr/src/app/main.cjs:27:3390)
    at sqliteTransaction (/usr/src/app/node_modules/better-sqlite3/lib/methods/transaction.js:65:24)
    at Object.C6i [as transactional] (/usr/src/app/main.cjs:963:1970)
    at LQe._setContent (/usr/src/app/main.cjs:27:3360)
    at LQe.setContent (/usr/src/app/main.cjs:943:3398)
JS Error: 500 PUT notes/LEb4xaWTrHVx/data - database is locked
Stack: Error
    at logError$1 (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:28:80574)
    at throwError (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:28:87126)
    at reportError (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:2:5999)
    at async Object.error (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:2:4424)
JS Error: Uncaught error: Message: 500 PUT notes/LEb4xaWTrHVx/data - database is locked, Line: undefined, Column: undefined, Error object: {}, Stack: Error: 500 PUT notes/LEb4xaWTrHVx/data - database is locked
    at throwError (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:28:87140)
    at reportError (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:2:5999)
    at async Object.error (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:2:4424)
Stack: Error
    at Object.logError$1 [as logError] (https://notes.nasreddine.com/assets/v0.100.0/src/app_context.js:28:80574)
    at https://notes.nasreddine.com/assets/v0.100.0/src/index_jquery.js:1:1384
Got error  SqliteError: database is locked
    at /usr/src/app/main.cjs:963:1214
    at Q5 (/usr/src/app/main.cjs:963:1591)
    at Ofe (/usr/src/app/main.cjs:963:1204)
    at Object.b6i [as upsert] (/usr/src/app/main.cjs:963:115)
    at LQe.saveBlob (/usr/src/app/main.cjs:27:4288)
    at sqliteTransaction.<anonymous> (/usr/src/app/main.cjs:27:3390)
    at sqliteTransaction (/usr/src/app/node_modules/better-sqlite3/lib/methods/transaction.js:65:24)
    at Object.C6i [as transactional] (/usr/src/app/main.cjs:963:1970)
    at LQe._setContent (/usr/src/app/main.cjs:27:3360)
    at LQe.setContent (/usr/src/app/main.cjs:943:3398) {
  code: 'SQLITE_BUSY'
}

Kubernetes manifest

apiVersion: v1
data:
  main.sh: |
    #!/usr/bin/env sh
    set -eo pipefail

    if [ "$#" -ne 1 ]; then
      echo "USAGE: $0 <database>" >&2
      exit 1
    fi

    echo "Attempting to restore the database from backup..." >&2
    if ! litestream restore -if-db-not-exists -if-replica-exists "$1"; then
      echo "Restoration failed, will assume it's not possible to replicate" >&2
    fi

    echo "Restoring is complete..." >&2
    touch /tmp/litestream-restored

    echo "Replication will start now..." >&2
    exec litestream replicate
kind: ConfigMap
metadata:
  name: litestream-scripts
  namespace: trilium
---
apiVersion: v1
data:
  litestream.yml: |
    dbs:
      - path: /home/node/trilium-data/document.db
        replica:
          type: s3
          bucket: trilium
          path: wael/db
          endpoint: https://api.minio.prod.nasreddine.com
          region: us-west-1
          sign-payload: true
kind: ConfigMap
metadata:
  name: litestream-wael
  namespace: trilium
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: trilium-wael
  namespace: trilium
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: freenas-iscsi-csi-immediate-xfs
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: main
    app.kubernetes.io/instance: trilium-wael
    app.kubernetes.io/name: trilium
  name: trilium-wael
  namespace: trilium
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app.kubernetes.io/component: main
      app.kubernetes.io/instance: trilium-wael
      app.kubernetes.io/name: trilium
  serviceName: trilium-wael
  template:
    metadata:
      annotations:
        reloader.stakater.com/auto: "true"
      labels:
        app.kubernetes.io/component: main
        app.kubernetes.io/instance: trilium-wael
        app.kubernetes.io/name: trilium
    spec:
      automountServiceAccountToken: true
      containers:
        - env:
            - name: SQLITE_BUSY_TIMEOUT
              value: "30000"
          image: triliumnext/trilium:v0.100.0
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 10
          name: trilium
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 10
          resources:
            limits:
              memory: 700Mi
            requests:
              cpu: 100m
              memory: 700Mi
          startupProbe:
            failureThreshold: 10
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 45
            periodSeconds: 10
            timeoutSeconds: 5
          volumeMounts:
            - mountPath: /home/node/trilium-data/config.ini
              name: config
              subPath: config.ini
            - mountPath: /home/node/trilium-data
              name: data
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      hostIPC: false
      hostNetwork: false
      hostPID: false
      initContainers:
        - args:
            - sh
            - -c
            - chown -R 1000:1000 /home/node/trilium-data
          image: busybox:stable
          name: fixperms
          resources:
            limits:
              memory: 32Mi
            requests:
              cpu: 100m
              memory: 32Mi
          securityContext:
            runAsGroup: 0
            runAsUser: 0
          volumeMounts:
            - mountPath: /home/node/trilium-data
              name: data
        - args:
            - /home/node/trilium-data/document.db
          command:
            - /scripts/main.sh
          env:
            - name: GODEBUG
              value: http2client=0
          envFrom:
            - secretRef:
                name: litestream-wael
          image: litestream/litestream:0.5.3
          name: litestream
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 2100m
              memory: 512Mi
          restartPolicy: Always
          startupProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - test -f /tmp/litestream-restored
          volumeMounts:
            - mountPath: /home/node/trilium-data
              name: data
            - mountPath: /etc/litestream.yml
              name: litestream-wael
              subPath: litestream.yml
            - mountPath: /scripts
              name: litestream-scripts
      serviceAccountName: default
      volumes:
        - configMap:
            name: trilium-wael-config
          name: config
        - configMap:
            name: litestream-wael
          name: litestream-wael
        - name: data
          persistentVolumeClaim:
            claimName: trilium-wael
        - configMap:
            defaultMode: 365
            name: litestream-scripts
          name: litestream-scripts
  updateStrategy:
    type: RollingUpdate

[dosubot[bot]]

Hi @kalbasit! I'm Dosu and I’m helping the Trilium team.

Trilium uses a single, long-lived SQLite connection (via better-sqlite3) and explicitly sets WAL mode, which is compatible in principle with Litestream's backup approach. However, Trilium does not set or honor the SQLITE_BUSY_TIMEOUT environment variable, so setting it has no effect in this setup. The "database is locked" errors occur because both Trilium and Litestream are accessing the database concurrently, and Trilium doesn't implement any special concurrency or locking logic beyond SQLite's defaults. If Litestream tries to access the database while Trilium holds a lock, or if Litestream doesn't back off on busy errors, Trilium can get stuck until Litestream is stopped or removed source.

To mitigate this, ensure Litestream is configured to only read the WAL and not open the database for writing, and that it is set to aggressively retry or back off on busy errors. If the locking persists, consider scheduling Litestream to run backups when Trilium is idle, or use periodic file-level backups instead of continuous streaming. There are no Trilium-specific workarounds or recent changes addressing this; the main compatibility consideration is careful coordination to avoid concurrent write access source.

If you need a more robust backup, Trilium's built-in backup feature creates snapshots in the data directory, which you can then sync to remote storage. This avoids the concurrency issues seen with continuous WAL streaming sidecars.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}