Prune all operations in a recurring task by markylaing · Pull Request #18516 · canonical/lxd

markylaing · 2026-06-22T16:05:09Z

Updates the operation prune task to run every minute. Operations with children are kept for 24 hours, but all other operations are deleted if they are complete and last updated more than 5 seconds ago.

This means that all operations will hang around for a bit longer, but none should be left lingering forever.

Closes #18475

Checklist

I have read the contributing guidelines and attest that all commits in this PR are signed off, cryptographically signed, and follow this project's commit structure.
I have checked and added or updated relevant documentation.

Copilot

Pull request overview

This PR adjusts LXD’s operation cleanup behavior to prevent completed operations from lingering indefinitely in the database under heavy load by moving deletion responsibility into the recurring prune task and running it more frequently.

Changes:

Run the expired-operation prune task every minute instead of hourly.
Update the prune logic to retain parent (bulk) operations for 24 hours, while pruning other completed operations after a short grace period (~5s).
Remove immediate DB deletion from individual operation completion in favor of centralized pruning.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
`lxd/operations/operations.go`	Changes in-memory operation cleanup on completion (removes prior delayed/DB cleanup goroutine).
`lxd/operations/linux.go`	Updates pruning criteria to differentiate parent (bulk) vs other operations and introduces 5-second grace window for non-bulk.
`lxd/operations.go`	Switches prune task scheduling from hourly to every minute.

github-actions · 2026-07-03T12:52:39Z

This PR has no domain label assigned. @canonical/lxd-maintainers please set the appropriate domain labels.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

markylaing · 2026-07-21T16:14:54Z

@tomponline @kadinsayani this is also now rebased and ready for review. There will be conflicts with operation stages so I'll fix up and rebase whichever one is merged second.

markylaing · 2026-07-22T15:56:12Z

+// This is the only place where operations are deleted from the in-memory map.
+// They are also deleted from the database at startup and shutdown, and when a member is offline.
+func SynchronizeAndPruneExpiredOperations(ctx context.Context, s *state.State) error {
+	now := time.Now()


We need to consider operation statuses both locally and in the database.

A good heuristic would be to look at the updatedAt timestamp for both and update those that don't match (or delete those that are not present).

Consider the local map as the source of truth.

markylaing · 2026-07-22T15:57:03Z

 			continue
 		}

+		// Skip operations that finished more than 5 seconds ago.


markylaing · 2026-07-22T15:59:46Z

  lxc operation list --project default

+  # Prune any leftover operations before starting the test.
+  lxc query -X POST /internal/testing/prune-operations


Should look at why the test is failing and make the test more precise

And don't spawn a goroutine or delete the operation from the internal map or database. Signed-off-by: Mark Laing <mark.laing@canonical.com>

… is readonly. This check is present for `op.{Update,Extend}Metadata` but not on commit. Signed-off-by: Mark Laing <mark.laing@canonical.com>

Delete in batches of 500 to avoid issues with SQLite max bind parameters. Signed-off-by: Mark Laing <mark.laing@canonical.com>

And update other unit tests to use the new function. Signed-off-by: Mark Laing <mark.laing@canonical.com>