feat(xo-server/backups): set the full listing interval of a repository in its URL

An entry of the VM backups cache is listed in full again an hour after it was built, to bound the
drift from the changes its journal does not record. The hour suits neither a large repository on an
object storage, where each listing reads the metadata of every VM, nor one whose backups are often
changed outside the journal.

The `fullListingInterval` parameter of the URL of a repository now sets this delay, as a duration
(`?fullListingInterval=6h`) or a number of milliseconds, read through `parseDuration()` like the
durations of the configuration. Without it, the delay is still an hour; with an invalid value too, with
a warning, so that a typo does not make the repository unreadable.

It is read when the entry is built: the URL changing already rebuilds the entry, which then reads the
new value.

Author: KAN-vates

CHANGELOG.unreleased.md

@@ -15,7 +15,7 @@
 - [Tasks] Add link and object resolution to Tasks Overview panel, better text flow for resolved task names (PR [#10265](https://github.com/vatesfr/xen-orchestra/pull/10265))
 - [XO5/Settings/IPs] Show an example of the expected IP format when adding IPs to an IP pool (PR [#10522](https://github.com/vatesfr/xen-orchestra/pull/10522))
 - [BRs] Add backup repository create form (PR [#10271](https://github.com/vatesfr/xen-orchestra/pull/10271))
-- [REST API] SSE now supports the `backup-archive` collection: backups appearing, being merged or disappearing from a backup repository are pushed to the subscribers, instead of each client listing the repositories again to spot them. They are sent as soon as a backup job or a deletion changes a backup repository, and within a couple of minutes for changes made outside this XO; a repository attached to a proxy which has not been updated yet is read every 10 minutes instead. When XO starts, it lists every enabled backup repository and sends the archives they already hold as `add` events; a repository it cannot reach then is sent the next time a client lists it (PR [#10472](https://github.com/vatesfr/xen-orchestra/pull/10472))
+- [REST API] SSE now supports the `backup-archive` collection: backups appearing, being merged or disappearing from a backup repository are pushed to the subscribers, instead of each client listing the repositories again to spot them. They are sent as soon as a backup job or a deletion changes a backup repository, and within a couple of minutes for changes made outside this XO; a repository attached to a proxy which has not been updated yet is read every 10 minutes instead. When XO starts, it lists every enabled backup repository and sends the archives they already hold as `add` events; a repository it cannot reach then is sent the next time a client lists it. Each repository is also listed in full every hour, for the changes its journal does not record; the `fullListingInterval` parameter of its URL changes that delay, e.g. `?fullListingInterval=6h` (PR [#10472](https://github.com/vatesfr/xen-orchestra/pull/10472))
 - [Plugins/load balancer] Added VM-to-host affinity to force VMs to run on a given set of hosts if possible (PR [#10207](https://github.com/vatesfr/xen-orchestra/pull/10207))
 - [Plugins/load balancer] Improved migration decision making (PR [#10207](https://github.com/vatesfr/xen-orchestra/pull/10207))
 - [New VM] Allow removing existing (template) disks (PR [#10292](https://github.com/vatesfr/xen-orchestra/pull/10292))

packages/xo-server/src/xo-mixins/backups-ng/_vmBackupsCache.mjs

@@ -5,6 +5,8 @@ import { compareTimestamp } from '@xen-orchestra/backups/RemoteAdapter.mjs'
 import { createLogger } from '@xen-orchestra/log'
 import { EventEmitter } from 'node:events'
 import { journalCursorAt } from '@xen-orchestra/backups/_backupJournal.mjs'
+import { parse } from 'xo-remote-parser'
+import { parseDuration } from '@vates/parse-duration'
 
 /**
  * A backup repository, as required by `VmBackupsCache`.
@@ -67,6 +69,8 @@ import { journalCursorAt } from '@xen-orchestra/backups/_backupJournal.mjs'
  * successfully once, i.e. whether it disappearing on the next replay is anomalous rather than benign
  * @property {Repository['options']} [options]
  * @property {string} [proxy]
+ * @property {number} rebuildInterval how long after `builtAt` the repository is listed in full again,
+ * see `rebuildIntervalOf()`
  * @property {number} refreshedAt when this process last brought the entry up to date: with `builtAt`,
  * the only fields which may be compared with its clock
  * @property {boolean} stale
@@ -87,8 +91,8 @@ const CLOCK_SKEW_TOLERANCE = 5 * 60 * 1e3
 
 // how long an entry is brought up to date from the journal before it is listed in full again, i.e. the
 // longest the drift described in `VmBackupsCache` lasts, at the price of one full listing of each
-// repository per period
-const REBUILD_INTERVAL = 60 * 60 * 1e3
+// repository per period, unless its URL says otherwise, see `rebuildIntervalOf()`
+const DEFAULT_REBUILD_INTERVAL = 60 * 60 * 1e3
 
 /**
  * @param {Entry} entry
@@ -98,6 +102,39 @@ const REBUILD_INTERVAL = 60 * 60 * 1e3
 const isSameRepository = (entry, repository) =>
   entry.url === repository.url && entry.options === repository.options && entry.proxy === repository.proxy
 
+/**
+ * How long an entry of a repository is brought up to date from the journal before it is listed in
+ * full again: the `fullListingInterval` parameter of its URL, a duration such as `6h` or a number of
+ * milliseconds, otherwise `DEFAULT_REBUILD_INTERVAL`.
+ *
+ * Read when the entry is built: the URL changing rebuilds the entry anyway, see `isSameRepository()`.
+ *
+ * @param {Repository} repository
+ * @returns {number}
+ */
+const rebuildIntervalOf = repository => {
+  const { fullListingInterval } = parse(repository.url)
+  if (fullListingInterval === undefined) {
+    return DEFAULT_REBUILD_INTERVAL
+  }
+
+  try {
+    return parseDuration(fullListingInterval)
+  } catch (error) {
+    if (!(error instanceof TypeError)) {
+      throw error
+    }
+
+    // a typo in an optional setting must not make the repository unreadable
+    warn('invalid fullListingInterval, using the default one', {
+      repositoryId: repository.id,
+      fullListingInterval,
+      defaultInterval: DEFAULT_REBUILD_INTERVAL,
+    })
+    return DEFAULT_REBUILD_INTERVAL
+  }
+}
+
 /**
  * forgets a backup, and the VM it belonged to when it was its last one
  *
@@ -279,7 +316,8 @@ export function serveVmBackups(backupsByVm, remoteId, vmId) {
  * How a repository is read is `VmBackupsSource`'s business: this class only decides *when* to list
  * it, when to replay it, and what to serve in the meantime.
  *
- * Entries are rebuilt from scratch an hour after they were built, which bounds the drift accumulated
+ * Entries are rebuilt from scratch an hour after they were built, or after the `fullListingInterval`
+ * of the URL of the repository, see `rebuildIntervalOf()`, which bounds the drift accumulated
  * from the events which could not be journaled, or which are not journaled at all (e.g. the
  * `immutable-backups` daemon lifting the immutability of a backup), when the remote is re-pointed,
  * reconfigured or moved to another proxy, which the entry detects by itself from what it was built
@@ -315,7 +353,8 @@ export class VmBackupsCache {
     first: async (/** @type {string} */ id) => this.#listArchives().find(archive => archive.id === id),
   }
 
-  // repository id → { backupsByVm, builtAt, cursor, journalConfirmed, options, proxy, refreshedAt, stale, url }
+  // repository id → { backupsByVm, builtAt, cursor, journalConfirmed, options, proxy, rebuildInterval,
+  // refreshedAt, stale, url }
   /** @type {Map<string, Entry>} */
   #entries = new Map()
 
@@ -519,7 +558,7 @@ export class VmBackupsCache {
     const now = Date.now()
 
     try {
-      if (entry === undefined || !isSameRepository(entry, repository) || now - entry.builtAt >= REBUILD_INTERVAL) {
+      if (entry === undefined || !isSameRepository(entry, repository) || now - entry.builtAt >= entry.rebuildInterval) {
         return await this.#build(repository)
       }
 
@@ -672,6 +711,7 @@ export class VmBackupsCache {
       cursor: journalCursorAt(now - CLOCK_SKEW_TOLERANCE),
       options: repository.options,
       proxy: repository.proxy,
+      rebuildInterval: rebuildIntervalOf(repository),
       refreshedAt: now,
       stale: false,
       url: repository.url,

packages/xo-server/src/xo-mixins/backups-ng/_vmBackupsCache.test.mjs

@@ -7,7 +7,7 @@ import { normalize } from '@xen-orchestra/fs/path'
 import { filenameOf, journalEntryPath, metadataOf, OTHER_VM, VM } from './_vmBackupsFixtures.mjs'
 import { serveVmBackups, VmBackupsCache } from './_vmBackupsCache.mjs'
 
-const REPOSITORY = { id: 'repository' }
+const REPOSITORY = { id: 'repository', url: 'file:///media/backup' }
 
 const enoent = () => Object.assign(new Error('ENOENT'), { code: 'ENOENT' })
 
@@ -319,6 +319,50 @@ describe('VmBackupsCache', () => {
     assert.equal(repository.nJournalReads, 1)
   })
 
+  it('rebuilds after the fullListingInterval of the URL of the repository', async t => {
+    const { tick } = mockTime(t, Date.parse('2026-08-11T10:00:00Z'))
+    const repository = new Repository([metadataOf(VM, '20260811T090000')])
+    const cache = new VmBackupsCache(repository.source, { minRefreshDelay: 60e3 })
+    const withInterval = { ...REPOSITORY, url: 'file:///media/backup?fullListingInterval=10m' }
+
+    await cache.get(withInterval)
+    tick(9 * 60e3)
+    await cache.get(withInterval)
+    assert.equal(repository.nListings, 1)
+
+    tick(60e3)
+    await cache.get(withInterval)
+    assert.equal(repository.nListings, 2)
+  })
+
+  it('reads a fullListingInterval in milliseconds', async t => {
+    const { tick } = mockTime(t, Date.parse('2026-08-11T10:00:00Z'))
+    const repository = new Repository([metadataOf(VM, '20260811T090000')])
+    const cache = new VmBackupsCache(repository.source, { minRefreshDelay: 60e3 })
+    const withInterval = { ...REPOSITORY, url: 'file:///media/backup?fullListingInterval=600000' }
+
+    await cache.get(withInterval)
+    tick(10 * 60e3)
+    await cache.get(withInterval)
+    assert.equal(repository.nListings, 2)
+  })
+
+  it('rebuilds an hour after the entry was built when the fullListingInterval is invalid', async t => {
+    const { tick } = mockTime(t, Date.parse('2026-08-11T10:00:00Z'))
+    const repository = new Repository([metadataOf(VM, '20260811T090000')])
+    const cache = new VmBackupsCache(repository.source, { minRefreshDelay: 60e3 })
+    const withInterval = { ...REPOSITORY, url: 'file:///media/backup?fullListingInterval=often' }
+
+    await cache.get(withInterval)
+    tick(59 * 60e3)
+    await cache.get(withInterval)
+    assert.equal(repository.nListings, 1)
+
+    tick(60e3)
+    await cache.get(withInterval)
+    assert.equal(repository.nListings, 2)
+  })
+
   it('rebuilds when the remote is re-pointed or reconfigured', async t => {
     mockTime(t, Date.parse('2026-08-11T10:00:00Z'))
     const repository = new Repository([metadataOf(VM, '20260811T090000')])

packages/xo-server/src/xo-mixins/backups-ng/index.mjs

@@ -751,8 +751,9 @@ export default class BackupNg {
   // listed since, and the timers of `#pollVmBackupArchives()` read all of them
   //
   // one after another rather than all at once: each listing reads the metadata of every VM of the
-  // repository; it also staggers their hourly rebuilds, which count from the listing, by as long as
-  // the listings take, i.e. the most for the repositories which are the slowest to list
+  // repository; it also staggers their periodic rebuilds (hourly unless the URL of the repository sets
+  // `fullListingInterval`), which count from the listing, by as long as the listings take, i.e. the
+  // most for the repositories which are the slowest to list
   //
   // through `_listVmBackupsWithBackoff()`, which never rejects: a repository which cannot be read then
   // is left out of the collection until a client lists it