add worker shutdown options

[fcollman]

No description provided.

[Copilot]

Pull request overview

This PR introduces a worker auto-shutdown mechanism for Celery workers, allowing them to gracefully terminate after processing a configurable number of tasks. This is useful for environments like Kubernetes where worker pods need to be recycled periodically to prevent memory leaks or apply configuration updates.

Key Changes

• Added three new configuration options to control worker auto-shutdown behavior (enabled flag, max tasks, and delay)
• Implemented a task post-run signal handler that tracks task execution count and triggers shutdown when the threshold is reached
• Integrated shutdown configuration into the Celery app creation process

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
materializationengine/config.py	Adds three new configuration variables for controlling worker auto-shutdown
materializationengine/celery_worker.py	Implements the auto-shutdown mechanism with task counting, signal handling, and graceful termination logic

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.