Skip to content

Fix updater self-termination during deb upgrades - #1430

Open
kassi0-peia wants to merge 5 commits into
ilysenko:mainfrom
kassi0-peia:fix/updater-deb-self-termination
Open

Fix updater self-termination during deb upgrades#1430
kassi0-peia wants to merge 5 commits into
ilysenko:mainfrom
kassi0-peia:fix/updater-deb-self-termination

Conversation

@kassi0-peia

Copy link
Copy Markdown

Summary

Prevent the Debian package prerm hook from stopping codex-update-manager.service when the package is being upgraded.

The updater can launch its own package upgrade through:

codex-update-manager -> pkexec -> apt -> dpkg

During that transaction, dpkg invokes the package prerm. The existing script unconditionally stops codex-update-manager.service, including for the Debian upgrade action. That causes the service to terminate the privileged installation it launched itself.

Observed failure

On Linux Mint, a normal automatic update repeatedly produced this sequence:

  • Polkit authentication succeeded.
  • apt / dpkg began installing the rebuilt codex-desktop package.
  • prerm stopped codex-update-manager.service.
  • systemd attempted to tear down the user-service cgroup.
  • root-owned apt / dpkg descendants remained behind.
  • the user manager logged Operation not permitted while trying to kill the cgroup.
  • the updater restarted and rebuilt/retried the same candidate again.
  • the user was prompted for authentication again.

The loop also repeatedly consumed several GB of RAM and significant CPU while rebuilding the package.

Fix

Return immediately from codex-update-manager.prerm for the Debian upgrade action.

Removal/deconfiguration behavior is unchanged, so actual package removal still stops/disables the user service.

This lets the running updater finish the package transaction. Its existing replacement detection can then observe that its on-disk binary changed and allow systemd to restart the updated daemon normally.

Validation

Verified locally with:

  • shell syntax checks for the package maintainer scripts
  • tests/scripts_smoke.sh — 57/57 tests passing
  • git diff --check
  • a full automatic signed upstream update, end-to-end

The automatic update completed successfully to upstream 26.901.41123, the updater reached installed, and there were no orphaned apt / dpkg processes or further self-termination/rebuild loop.

A smoke-test assertion was added so the Debian upgrade guard cannot be removed accidentally.

@ilysenko ilysenko left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the Debian updater lifecycle issue and documenting the reproduced failure. Two blockers remain:

  1. Skipping service shutdown during prerm upgrade leaves the old daemon running after /usr/bin/codex-update-manager is replaced. The current updater has no replacement detection, postinst does not restart an active unit, and the next rebuild receives a current_exe() path ending in (deleted). Please preserve the in-flight package transaction, then safely exit/restart the daemon only after the install result and Installed state have been persisted.
  2. The added smoke assertions only check for strings and still pass if the actual exit 0 guard is removed. Please add behavioral coverage for the staged DEBIAN/prerm: upgrade must avoid service side effects, removal/deconfiguration must retain cleanup, and a successful self-upgrade must restart onto the new binary and allow the next rebuild.

Please run the complete updated base-to-head diff through your own code-review model, fix every blocker it finds, rerun the review, and repeat until that model reports no remaining blockers before requesting maintainer re-review.

The Debian prerm script unconditionally stopped codex-update-manager.service, including when invoked for an upgrade.

When an automatic update was launched by codex-update-manager itself, the service spawned pkexec -> apt -> dpkg and dpkg then ran the package prerm. Stopping the originating user service from that prerm interrupted its own privileged installation.

That left root-owned apt/dpkg processes in the user-service cgroup, produced repeated "Operation not permitted" cleanup failures, and caused the updater to restart and rebuild the same candidate again.

Skip user-service shutdown for the Debian upgrade prerm action. Removal and deconfiguration paths retain the existing stop/disable behavior. The running daemon can then finish the package transaction, detect that its on-disk updater binary was replaced, and let systemd restart the updated daemon normally.

Add smoke-test assertions covering the upgrade guard.

Observed before the fix:
- successful polkit authentication followed by updater service termination
- orphaned apt/dpkg processes remaining in the user-service cgroup
- repeated rebuild/password cycles
- multi-GB memory and CPU use on every retry

Validated after the fix:
- shell syntax checks pass
- tests/scripts_smoke.sh passes all 57 tests
- git diff --check passes
- a full automatic signed upstream update completed end-to-end
- updater reached status installed
- no orphaned apt/dpkg processes or self-termination loop remained
Preserve Debian self-upgrade transactions by leaving the updater service running from prerm during package upgrades, while retaining normal cleanup for removal and deconfiguration.

Detect executable replacement by device/inode. After a successful managed install, persist and read back Installed state before exiting with the Restart=on-failure code, allowing systemd to relaunch the new binary. Harden subsequent rebuilds against /proc/self/exe paths carrying the kernel's " (deleted)" suffix.

Exercise the staged Debian maintainer script behavior and a real old-inode/new-binary replacement lifecycle.

This restores the replacement-restart lifecycle originally introduced in 28f03cf and lost during the signed-package refactor 47f6d69; the changelog continued to describe this behavior.
@kassi0-peia

Copy link
Copy Markdown
Author

Addressed the updater self-termination and stale-binary lifecycle issues from the review.

The Debian prerm now treats upgrade differently from removal/deconfiguration: an upgrade leaves the running updater service alone so an updater-initiated apt/dpkg transaction cannot terminate its own parent service mid-install. remove and deconfigure retain the existing stop, disable, and daemon-reload cleanup.

The updater now detects executable replacement by device/inode rather than content. This handles the normal /proc/self/exe -> ... (deleted) case and also catches byte-identical package replacement. After a managed install succeeds, the updater persists Installed state and reads it back before exiting with the service restart code; systemd then relaunches the new binary. If persisted state is still Installing, replacement restart is blocked rather than exiting early.

The builder path is also hardened so a subsequent rebuild does not use a stale current_exe() path ending in " (deleted)"; it resolves the live replacement updater instead while preserving the healthy current executable for normal dev/test processes.

Behavioral coverage added for the requested cases:

  • staged DEBIAN/prerm upgrade produces no user-service side effects;
  • staged remove and deconfigure retain stop/disable/reload cleanup;
  • successful self-replacement runs the old inode, atomically replaces the updater, exercises the production restart exit path, starts the replacement binary, and verifies the next rebuild uses the live updater.

I also mutation-tested the Debian regression harness by removing the upgrade guard from a temporary staged prerm; the test then observed the forbidden systemctl --user stop codex-update-manager.service side effect, confirming the coverage is behavior-sensitive rather than string-only.

I ran repeated adversarial review passes over the updated diff and fixed the issues they exposed, including production-script staging in the test path, healthy-current-vs-installed binary resolution, exercising the real restart exit path, non-root CI skip behavior, and test-shell quoting.

Validation on the rebased branch:

  • cargo test -p codex-update-manager: 60/60 passing
  • repeated full updater suite: 10/10 passing before rebase finalization, plus 15/15 in the final paranoid pass
  • repeated self-replacement + staged Debian prerm: 50/50 combined paranoid rounds, in addition to the earlier 25/25 runs for each
  • updater Clippy with -D warnings: 3/3 earlier plus 5/5 final paranoid passes
  • script smoke suite: 3/3 earlier plus 5/5 final paranoid passes
  • Core CI: passing
  • Debian package CI: passing
  • RPM package CI: passing
  • Pacman package CI: passing
  • record/replay tests and Clippy: passing
  • final git diff --check: clean

The branch is rebased onto current origin/main and the worktree is clean.

@kassi0-peia
kassi0-peia force-pushed the fix/updater-deb-self-termination branch from f6e0dbe to 64ed90a Compare September 6, 2026 03:01

@ilysenko ilysenko left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough lifecycle fix. The original stale-daemon path and the staged prerm behavior are substantially improved, but three blockers remain:

  1. passive_replacement_disposition() blocks forever whenever the persisted status is Installing. If a separate install-ready or rollback process replaces the updater and then dies before saving Installed, the still-running service daemon will see the replacement every 15 seconds but will neither recover the interrupted transaction nor restart onto the installed binary. Please add bounded, ownership-aware recovery (for example, a lock/lease/liveness signal) so an active installer is protected while an abandoned Installing state can converge safely.
  2. tests/deb-prerm.test.js chooses /run/user/${40000 + pid % 20000} without proving that the directory was created by the test. Both privileged branches accept a pre-existing path; the sudo branch chmods it to 0777, and cleanup recursively removes the whole directory. A UID collision can therefore modify or delete another user's active runtime directory. Please reserve a guaranteed-new numeric directory atomically, track/validate that it belongs to this fixture, and clean up only what the test created.
  3. successful_self_replacement_restarts_on_new_binary_and_next_build_uses_it does not exercise the production install lifecycle: the child calls exit_for_replacement() directly on a marker and the parent manually starts the replacement. It never runs install_ready(), observes the package result, or proves the required save Installed -> read back -> exit ordering. Moving or deleting the production restart_after_persisted_install() call would leave this test green. Please add behavioral coverage through the production lifecycle so that exact ordering regression is caught.

Please run the complete updated base-to-head diff through your own code-review model, fix every blocker it finds, rerun the review, and repeat until that model reports no remaining blockers before requesting maintainer re-review.

Make updater replacement and abandoned-install recovery fail closed and ownership-aware.

- persist PID/start-time transaction ownership before privileged package mutation
- gate pkexec launch until package-command ownership is durably saved
- reconcile abandoned Debian installs only from fully configured package state
- reject ambiguous same-version and unknown pre-install recovery evidence
- preserve pre-transaction installed version for recovery and rollback bookkeeping
- exercise the production install/restart/readback lifecycle in replacement tests
- harden Debian prerm runtime-dir fixtures with verified fixture ownership
- make executable replacement fixtures deterministic under parallel test execution
Avoid publishing the long-lived updater daemon as the owner of a package transaction before the privileged command can start.

Persist the exact gated child identity before releasing package mutation, distinguish failures that are provably pre-mutation from ambiguous outcomes, and clear failed pre-launch transactions so a running daemon cannot remain stuck in Installing.

Also make abandoned-install grace fail toward recovery when the persisted wall-clock start time is in the future, preserving the bounded recovery guarantee across clock rollback, with regression coverage for both cases.

Copy link
Copy Markdown
Author

Since the last review, I reworked the updater replacement/recovery path around the three issues you called out and then did another adversarial pass over the resulting state machine.

The main changes are:

  • Added a gated package-command ownership layer that persists the exact child PID/start-time identity before privileged mutation is released, instead of publishing the long-lived updater daemon as the transaction owner.
  • Make abandoned Installing recovery bounded and ownership-aware, including future/rolled-back wall-clock timestamps.
  • Reconcile abandoned Debian installs only from strong package evidence: fully configured package state, matching artifact identity, matching candidate version, and an observed install effect.
  • Preserve the pre-transaction installed version so update and rollback recovery can reproduce the same bookkeeping as their successful paths.
  • Exercise the real install_ready() replacement lifecycle in the self-replacement tests, including Installed persistence/readback before exit and restart.
  • Harden the Debian prerm fixture so it only creates and cleans up runtime directories it can prove it owns.
  • After the gated package command has been released, preserve Installing plus the durable transaction on a later nonzero exit because package mutation may already have occurred. The same rule now applies to rollback.
  • Treat Debian prerm failed-upgrade like upgrade so upgrade failure/unwind cannot stop the updater that owns the package transaction.

I also added regressions for the new ownership/recovery cases, the production replacement lifecycle, post-release install/rollback failures, and the Debian failed-upgrade path.

Preserve durable install recovery evidence when a gated package command exits unsuccessfully after mutation may have started, including rollback.

Keep the updater service alive through Debian failed-upgrade unwind paths and add regressions for both cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants