Repository navigation
Conversation
KAN-vates
force-pushed
the
refactor/backups/vm-backup-archives-collection
branch
from
October 12, 2026 04:54
1c6d4ff to
622b5ed
Compare
… as an XO collection The REST API special-cased backup-archive to reach the cache's own emitter. The archives now provide what registerCollection expects, and the backups mixin registers them like any other collection. What the archives emit changes with it: one event per kind of change, with the archives added or updated, or the ids removed, instead of one event per archive with its previous value, which registerCollection keeps itself. A listener which throws is still caught, now once per kind: the listeners after it miss that kind's whole batch rather than one archive. The only listener in production, registerCollection's, queues each object and does not throw. The `stop` hook which removed every listener of the archives is dropped on purpose: registerCollection adds them, and keeps them for the life of the process, as it does for every collection.
The archives were only announced while a repository was being read, and it was only read when a client listed it: an SSE subscriber learnt about the backups created, merged or deleted at the next listing by any client. The repositories whose archives are in the collection are now read on a timer: - every minute for those whose journal can be replayed: it costs a journal read, never a full listing, and brings the changes of this process as well as those it does not make, such as the size the merge worker rewrites after the job, or another XO sharing the repository; - every 10 minutes for those whose journal cannot be read, e.g. attached to a proxy which does not expose it: each read lists them in full, i.e. reads the metadata of every VM. Both timers go through the listing backoff, so an unreachable repository is not read on every tick. An entry is also listed in full again an hour after it was built, rather than when it crosses a UTC day: now that the archives are pushed rather than read on demand, this bounds to an hour the drift from the changes the journal does not record. Counting from the build, rather than from the start of the hour, spreads these listings instead of reading every repository at once.
The hourly rebuild, which repositories each timer reads, the retry delay a failing read keeps, and a repository moved off a proxy while its journal is being read.
…kup archives at once After a backup job or a deletion, `refresh()` merely marked the entry of the repository stale: its archives were only announced at the next read, by the timer or by a client. A repository is now read as soon as this XO changes it: - when the export of a VM to it ends, from the logs of the worker, of the no-worker mode and of a proxy alike: its metadata, journal entry and retention are written by then, and so is clean-vm for an incremental backup; a full backup runs clean-vm after the export, so what it changes is read at the end of the job; - at the end of the job and after a deletion, which already called `refresh()`. Only when its journal can be replayed: a repository which can only be listed in full is left to its 10-minute timer. A read in progress when `refresh()` is called now reads again before it settles, as many times as `refresh()` is called meanwhile: it may have missed the mutation, and with several VMs finishing close to each other, a push would otherwise often be served the read which started just before. Not when reading again means listing the repository in full: with VMs finishing faster than a listing, the read would never settle; the repository stays stale, and the next read lists it.
…chives changes A deletion and the export of a VM announced without waiting for a listing, a repository not in the collection left alone, and a `refresh()` during a replay or a listing reading again, as many times as it is called, unless reading again means listing the repository in full.
KAN-vates
force-pushed
the
refactor/backups/vm-backup-archives-collection
branch
from
October 12, 2026 05:33
622b5ed to
ff46d4c
Compare
KAN-vates
marked this pull request as ready for review
October 12, 2026 07:13
The archives collection only held the repositories a client had listed since the process started, and the timers only read those: after a restart, an SSE subscriber received nothing from a repository until a client happened to list it. Every enabled repository is now listed once XO has started, so the collection holds all of their archives, announced as `add` events, and the timers read all of them from then on. - On the `started` event rather than in a `start` hook: the start hook of the remotes mixin invalidates the listing of every repository, which would drop what a listing started before it has read instead of announcing it. - One repository after another rather than all at once: each listing reads the metadata of every VM of the repository, and listing them in sequence also staggers their hourly rebuilds, which count from the listing, by as long as the listings take. - Through `_listVmBackupsWithBackoff()`: a repository which cannot be reached then is left out of the collection until a client lists it, as before. - Stopped at the next repository when XO stops.
…y in its URL An entry of the VM backups cache is listed in full again an hour after it was built, to bound the drift from the changes its journal does not record. The hour suits neither a large repository on an object storage, where each listing reads the metadata of every VM, nor one whose backups are often changed outside the journal. The `fullListingInterval` parameter of the URL of a repository now sets this delay, as a duration (`?fullListingInterval=6h`) or a number of milliseconds, read through `parseDuration()` like the durations of the configuration. Without it, the delay is still an hour; with an invalid value too, with a warning, so that a typo does not make the repository unreadable. It is read when the entry is built: the URL changing already rebuilds the entry, which then reads the new value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… as an XO collection
The REST API special-cased backup-archive to reach the cache's own emitter. The archives now provide what registerCollection expects, and the backups mixin registers them like any other collection.
Description
Short explanation of this PR (feel free to re-use commit message)
Checklist
Fixes #007,See xoa-support#42,See https://...)Introduced by <commit|PR>CHANGELOG.unreleased.md@xen-orchestra/lite/CHANGELOG.mdCHANGELOG.unreleased.md### Screenshotssection above this checklistReview process