[feature:gsoc26] Add generic_notification reminders and failure-needs-attention alerts for persistent mass upgrades

[Eeshu-Yadav]

Is your feature request related to a problem? Please describe.

The firmware upgrader doesn't have any notifications today - real-time updates flow only through WebSocket publishers, which mean nothing to an admin who isn't actively watching the batch page. A persistent batch with 5 devices still waiting 60 days later produces zero signal to anyone.

I want two distinct flows on top of openwisp-notifications:

• A periodic reminder every ~60 days for batches that still have pending children, so admins notice long-running stragglers.
• A failure-needs-attention notification for cases that actually need a human (the device was deactivated mid-pending, or the device was reachable but the flash itself failed).

They have to be distinguishable, otherwise routine offline-pending cycles drown the real "something is wrong" cases in noise.

Describe the solution I would implement

I would like to introduce both notification flows on top of sub-issue 04's retry pipeline.

1. Add a send_pending_upgrade_reminders Celery Beat task that runs the query BatchUpgradeOperation.objects.filter(status="in-progress", upgradeoperation__status="pending").distinct() (the .distinct() is required to dedupe the JOIN). For each qualifying batch, fire notify.send(sender=batch, type="generic_message", target=batch, description=...) with a description that includes the pending device count. The target=batch arg drives recipient routing (notification goes to the batch's organization's admins + all superusers per openwisp_notifications/handlers.py:71–137) and gives the bell-icon link a destination - the batch's admin change page. Passing url=... in additional kwargs lets me override that with the ?status=pending-filtered admin URL.

2. Add a last_reminder_at = DateTimeField(null=True, blank=True) to AbstractBatchUpgradeOperation via migration 0019_batchupgradeoperation_last_reminder_at.py (kept separate from sub-issue 01's 0018_* so persistence can ship without waiting on notifications). The reminder task fires for a batch when coalesce(last_reminder_at, created) <= now() - reminder_period - so the first reminder fires 60 days after batch creation (when last_reminder_at is still NULL the fallback is created), and subsequent reminders fire 60 days after the previous send. On send, update last_reminder_at = timezone.now(). Without the coalesce, a brand-new batch with pending children would fire a reminder on the next weekly scan instead of waiting the full 60 days.

3. Add the failure-needs-attention notification as a post_save signal handler on UpgradeOperation, connected from apps.py alongside the other signal handlers. Fire notify.send() when:

   • instance.is_persistent is True,
   • the new status == "failed",
   • the previous status wasn't already "failed" (track via Django's from_db() classmethod or stash _previous_status on the instance in __init__, so re-saves don't fire duplicates).

4. The handler covers both failure paths uniformly:

   • Device deactivated mid-pending: retry_pending_upgrade (sub-issue 04) sets status="failed" - transition pending → failed.
   • Non-RecoverableFailure terminal failure inside upgrade() (ReconnectionFailed, checksum error, generic Exception): the existing exception arms at base/models.py:971–999 set status="failed" - transition in-progress → failed.
   • Routine offline → pending → retry → pending cycles never reach status="failed", so the handler stays silent. That separation is exactly what we want - those go to the periodic reminder instead.

5. Two configurable settings:

   Setting	Default	Purpose
   ..._PERSISTENT_REMINDER_PERIOD	5184000 (60 days)	Per-batch reminder cadence
   ..._REMINDER_SCAN_PERIOD	604800 (7 days)	How often send_pending_upgrade_reminders runs

6. Use notify.send() without an explicit recipient kwarg - the default openwisp-notifications routing delivers to all organization administrators and superusers for the target's organization (via openwisp_notifications/handlers.py:71–137), which is exactly the audience we want.

7. The built-in "generic_message" notification type at openwisp_notifications/types.py:20–32 covers what we need (info-level, web-only, generic verb) with no pre-registration. The generic notification requirement in [feature] Persistent Mass Upgrades #379 maps to this built-in type. The actual type string passed to notify.send() is type="generic_message". No register_notification_type() call needed in apps.py.

8. Unit tests covering: no qualifying batches → no notification; qualifying batch produces exactly one notification with the right description/target/URL; cadence guard prevents duplicate fires within the 60-day window; advancing time past the window produces the next reminder; deactivated-device path fires the failure notification; non-RecoverableFailure retry path fires the failure notification; routine pending → pending cycles do NOT fire it. notify.send is mocked across all tests.

[nemesifier]

Updated this issue to align it with the current proposal tracking:

• Renamed field/API/form references from persistent to is_persistent where applicable.
• Pointed the generic notification wording to the built-in generic_message type where applicable.

To review the body diff, navigate to the top bar of the issue, click on "last edited by nemesifier", then click on "most recent". GitHub does not provide a quick URL for this view.