Codex bundled-marketplace refresh leaks staging directories and can consume tens of gigabytes

[neverdie88]

Codex Desktop repeatedly leaves abandoned bundled-marketplace staging directories under:

~/.codex/.tmp/bundled-marketplaces/openai-bundled.staging-*

This caused severe disk exhaustion on macOS:

• Abandoned staging directories: 323
• Space consumed: approximately 31 GiB
• Data-volume free space before cleanup: approximately 240 MiB
• Free space after cleanup: approximately 31 GiB

Root-cause evidence

The desktop updater:

1. Creates a new UUID staging directory for each reconciliation.
2. Copies bundled plugins concurrently using Promise.all and macOS ditto.
3. Starts recursive cleanup when the first copy or post-processing operation rejects.
4. Does not necessarily wait for the remaining copy processes to stop writing.
5. Performs recursive removal without retry/backoff.
6. Does not sweep abandoned openai-bundled.staging-* directories during startup.

The abandoned directories were partially cleaned: manifests were missing, while different subsets of copied plugins remained. This is consistent with cleanup racing outstanding copy operations.

Each retained directory was approximately 100 MiB:

31 GiB / 323 ≈ 98 MiB per directory

After the original cleanup, the problem reproduced live. Five new staging directories totaling approximately 503 MiB appeared within several minutes, despite approximately 33 GiB of available disk space. Therefore, low disk space increases the likelihood of failure but is not required to trigger the leak.

Additional contributing issues

• Failed reconciliation is retried with a new UUID directory.
• There is no filesystem/interprocess lock.
• Existing serialization appears process-local.
• Startup cleanup targets canonical marketplace directories, not staging siblings.
• Promotion deletes the active target before renaming staging, creating a failure window.
• There is no maximum-age or maximum-size policy.
• Telemetry does not report retained staging count, total bytes, or oldest age.

Suggested fixes

• Wait for every copy worker to settle before cleanup.
• Cancel and reap outstanding ditto processes on failure.
• Retry recursive removal with backoff and verify that the directory disappeared.
• Add an interprocess marketplace-refresh lock.
• Sweep stale *.staging-* directories during startup.
• Keep the existing marketplace until the replacement is fully staged and validated.
• Add staging age/size limits and abnormal-growth telemetry.
• Add explicit handling and reporting for ENOSPC.
• Test copy failure, cancellation, process restart, concurrent refresh, and low-disk scenarios.

Proposed implementation

Use a single interprocess lock for each managed marketplace and treat staging as a recoverable transaction:

1. Acquire an exclusive lock such as:

   ~/.codex/.tmp/bundled-marketplaces/openai-bundled.refresh.lock
   

   If another refresh owns the lock, reuse or wait for that refresh instead of creating another staging directory.

2. After acquiring the lock, remove abandoned openai-bundled.staging-* directories older than a conservative threshold. Never remove the directory owned by the active refresh.

3. Create one staging directory and record its ownership and start time. Check that available disk space is sufficient for the bundled marketplace plus a safety margin before copying.

4. Copy plugins with bounded concurrency. If any worker fails, cancel the remaining work and wait for every spawned ditto process to exit before attempting cleanup. Promise.allSettled can be used to ensure that all workers have finished; the first meaningful error should still be preserved for reporting.

5. Validate the completed staging tree before promotion:

   • Marketplace manifest exists and parses successfully.
   • Every declared plugin directory exists.
   • Every plugin contains a valid .codex-plugin/plugin.json.
   • The copied plugin count matches the filtered marketplace manifest.

6. Promote without first destroying the working marketplace:

   openai-bundled            -> openai-bundled.previous
   openai-bundled.staging-*  -> openai-bundled
   

   If the second rename fails, restore openai-bundled.previous. Delete the previous version only after the new marketplace is validated and active.

7. In finally, wait for all child processes and remove staging with retry/backoff for transient EBUSY, ENOTEMPTY, EPERM, EMFILE, and ENFILE failures. Verify that the directory no longer exists before considering cleanup successful.

8. Run the same stale-staging sweep during Codex startup so crashes, forced termination, and system restarts cannot leave permanent data behind.

9. Add a safety policy and telemetry:

   • Maximum staging age, for example 24 hours.
   • Maximum retained staging count, for example one inactive directory.
   • Maximum total staging size with a warning before the limit is reached.
   • Metrics for staging count, total bytes, oldest age, refresh duration, failure phase, cleanup retries, and cleanup outcome.
   • A user-visible warning when cleanup fails or available disk space is insufficient.

The essential ordering is:

lock -> sweep stale staging -> stage -> wait for all copies -> validate
     -> atomic promotion with rollback -> verified cleanup -> unlock

This removes the cleanup-versus-copy race, prevents concurrent refresh duplication, preserves the last working marketplace, and guarantees recovery after application termination.

Recovery performed

After confirming that the staging root contained only real directories matching the expected pattern and had no open handles, I removed only:

~/.codex/.tmp/bundled-marketplaces

I recreated the empty parent directory. Sessions, state databases, memories, skills, plugins, configuration, credentials, and project data were preserved.

Environment

• Codex Desktop: 26.715.72359
• Build: 5718
• macOS: 26.6.1 (25G76)
• Visible Codex windows during investigation: 1
• Live recurrence observed: September 1, 2026, 22:32–22:40 AEST

The precise operation that first rejected could not be recovered because the relevant updater error was not retained locally. However, the staging lifecycle defects and live recurrence were directly verified.