Reduce the up to 5 day delay in emailing alerts on new applications down to late morning of next day

[ianheggie-oaf]

Problem:

Currently Most users receive email alerts within 1-3 days of an application being submitted, but in worst-case scenarios it could take up to 5 days! This is principally due to uncoordinated random spreading of processing load of the three main steps.

Describe the solution you'd like

This proposal should change that to most users receiving email alerts between 11 PM the same day and 11 AM the next day after an application has been entered..

Additional context

Currently

1. morph at 10:00 UTC (8:00 PM AEST, 9:00 PM ADST) randomly spreads the processing of scrapers across the next 24 hours
2. planning alerts at 12 pm Australia/Sydney randomly spreads the importing of those results into planning alerts across the next 24 hours
3. planning alerts at 12 pm Australia/Sydney randomly spreads the sending of email alerts across the next 24 hours

Technically it spreads it across 24 hours * (count - 1) / (count) but the count of scrapers / authorities / email alerts are high enough its of little consequence.

Ideas for discussion

1. Spread the running of scrapers actively used by planning alerts randomly across 9pm to 9am (when few applications are entered) - this has multiple benefits:

• Halve the average scraping delay 12 to 6 hours since most of the applications are entered in the Peak times: 10am-2pm (lunch/work hours), 6-8pm (after work)
• Reduce load on council sites

2. Use the existing Webhooks system with urls like http://planningalerts/has_run// to planning alerts is told when a scraper has completed. Considerations:

• Sanitize the input - ignore if we are not paying attention to that scraper, ignore if we have already been advised in the last 12 hours
• Schedule the import of records say randomly in the next 2 hours - much much better than spread over 48 hours

3. Schedule the processing of alerts randomly in the next 2 hours after import.
4. Lock this down to only scrapers that admin has flagged as critical

I believe this also shifts background load of the planning alerts server from randomly across the day to a 12 hour period that has less load from web traffic.

[ianheggie-oaf]

@benrfairless - a candidate for external team?