SEP-1551: Derive Celery include and beat seed prefixes from the app registry

[marcuscruz-percona]

Summary

• Apps declare their Celery task module via App.celery_module_path, a three-state field mirroring api_router_path (auto-derive <module>.celery from a filesystem probe, explicit string override, or explicit null opt-out); the worker include list and beat task_name prefixes both derive from that single source, so a module rename can no longer desync the include from the seed.
• AppRegistry exposes the ordered app-owned Celery module paths; build_celery_include composes static base + registry-derived app modules and is the one seam both the Celery-app assembly and the worker bootstrap call.
• A system periodic-task schedule is emitted only when its owning app contributes a Celery module — an absent or opted-out app yields no schedule instead of a None-prefixed task_name; an explicit override is filesystem-probed at settings construction so a typo fails at load, not at worker import.

Deviation from AC (flagged): CeleryOptions.set_include seeds only the static base rather than composing the app modules itself. Reaching the app registry during core Settings construction forces sep_settings, which reads core settings back through the same un-guarded lazy proxy and re-enters a half-built Settings. Composition therefore happens at a safe seam (build_celery_include) after core settings is fully built; nothing consumes Celery config before app.celery is imported, which is where composition runs.

Tested

• Targeted regression: config, registry, celery_registration, seed, settings-override integration, main — 160 passed.
• Runtime smoke: settings.CELERY.include == build_celery_include() == celery.conf.include == ['app.tasks.celery', 'app.sep.apps.snippets.celery', 'app.sep.apps.alerts.celery', 'app.sep.apps.report.celery'].
• make checkmigrations clean (no migration; beat rows re-seed by stable name).

Checklist

• New/modified functions have type hints and rST docstrings
• New tests added for new features or bug fixes
• All tests pass locally (make test) — ran targeted regression (160 passed), not the full suite
• Pre-commit hooks pass (make run-pre-commit)
• Database migrations generated if models changed (make makemigrations) — N/A, no model change
• User-facing changes documented (README, inline help, UI text) — N/A
• Configuration changes documented with examples — App.celery_module_path documented in the field docstring
• Changelog fragment added under changelog.d/

[Copilot]

Pull request overview

This PR centralizes SEP’s Celery module discovery by introducing App.celery_module_path and deriving both the worker include list and SEP beat seed task_name prefixes from the same app-registry-derived source, preventing drift when app task modules move/rename.

Changes:

• Add App.celery_module_path (three-state: omitted/explicit string/explicit null) with filesystem-based validation during settings construction.
• Introduce registry helpers (app_celery_module_paths, app_celery_module_for, build_celery_include) and plumb them into Celery app assembly, worker bootstrap, and SEP beat seeding (including “no celery module → no schedule” gating).
• Add/extend tests covering derivation, gating behavior, and invariants; add a changelog fragment.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
app/core/celery/config.py	Introduces STATIC_CELERY_INCLUDE and narrows CeleryOptions.set_include to seed the static base only.
app/celery.py	Composes the final settings.CELERY.include via build_celery_include() at Celery app assembly time.
app/main.py	Updates worker bootstrap to use build_celery_include() instead of a hardcoded include list.
app/sep/config.py	Adds App.celery_module_path plus convention-based defaulting and filesystem existence validation.
app/sep/apps/framework/registry.py	Exposes ordered app celery module paths and provides build_celery_include() for shared composition.
app/sep/db/seed.py	Seeds app-owned periodic schedules only when the owning app contributes a Celery module path; refactors report schedules into a helper.
tests/app/sep/test_config.py	Adds coverage for the celery_module_path three-state behavior, including override validation and opt-out.
tests/app/sep/apps/framework/test_registry.py	Adds tests for registry-derived celery module lists and include composition.
tests/app/sep/db/test_seed.py	Adds tests ensuring schedules are omitted when a plugin is absent or opts out of Celery.
tests/app/sep/test_celery_registration.py	Extends invariant tests to assert include/seed prefixes derive from the registry.
changelog.d/SEP-1551.changed.md	Documents the behavioral change for Celery include list and seed prefix derivation.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
app
celery.py
main.py
app/core/celery
config.py
app/sep
config.py					221
inventory.py
app/sep/apps/framework
registry.py
app/sep/apps/mysql_backups/restore
deps.py
app/sep/db
seed.py					116-130
app/sep/sync/syncers
pmm.py
Project Total

This report was generated by python-coverage-comment-action

[yyyyyyyan]

@marcuscruz-percona — solid refactor. build_celery_include gives the include list one derivation that both app/celery.py and start_celery_worker compose, the test_configured_include_is_the_derived_single_source / test_app_owned_seed_prefixes_track_registry_modules invariants pin the seed↔include↔registration contract structurally, and the new App.celery_module_path three-state field faithfully mirrors api_router_path — the explicit-override branch even adds a fail-fast filesystem probe the router field defers to startup. The circular-re-entry deviation is clearly flagged in the description and the reasoning holds: nothing reads settings.CELERY.include before app.celery reassigns it. Validator ordering, path arithmetic, and the None-prefixed-task_name guard rails all check out.

Three Minor, non-blocking notes inline:

• AppRegistry.celery_module_paths looks unused by production — every consumer calls the standalone helpers, never the registry property.
• The on-disk module-existence probe is duplicated verbatim between the two config.py validators.
• The _alerts_plugin_enabled / _report_plugin_enabled gates in the seed are redundant with app_celery_module_for(...) and inconsistent with the snippets path.

Nothing blocking — approving. Happy to re-review if you rework any of these.