Texter loading process freezes on various Spoke campaigns

[ErikKirkstein]

Over the past week or so, we are increasingly seeing challenges with Spoke loading all texts to assigned texters. In many cases, the texter loading screen will remain stuck at 0% or 20% or X% for upwards of 10 minutes, and in some cases indefinitely.

This issue came to head on Saturday, 9/29/18, where we were assigning texts into 37 campaigns, and about 10 of them froze up at least once, some froze many times resulting in admins having to basically ignore them to avoid disruptions in texting.

When I use the "discard job" feature to cancel the texter loading to reset the texter loading process, sometimes the texters assigned before the freeze up will receive their text request, other times they will appear in the texter menu but with zero texts. In some cases, when we attempt to add the texter or texts to a texter in that same campaign again, the texter loading screen will again remain stuck at 0% or 20%. And we then start this process over. This issue happens at all times of day, regardless of how many campaigns are open and how many texters are currently texting at a given time.

This is causing the following.

1. Some texters do not receive their requested texts
2. Texter confusion, whether they don't get their assigned texts or end up with more than assigned, leading to increased admin time for myself (trying to unlock froze up Spoke campaigns) and for volunteer leaders (trying to determine which campaign they were assigned to that didn't assign the requested texts, or having to ping me to unlock frozen campaigns).
3. Because there are often many campaigns live in Spoke, and we are receiving many text requests, we have to try to assign these texters to other campaigns only to find that later the texts assigned to the stuck Spoke campaign sometimes end up appearing, which leads to texters getting more texts than requested. This is resulting in some texts not being sent as the texter does not notice these additional texts
4. When we try and remove the surplus texts from a texter, the texter loading freezes again in Spoke campaigns and we basically cannot remove them from a campaign.

Due to the volume of texting our team is currently doing on a daily basis, and that this volume is only going to increase in the coming days and weeks, we are finding ourselves unable to keep track of how many campaigns freeze up and which texters were assigned to the campaigns that froze in order to attempt reassign them to other Spoke campaigns if/when it's possible.

From a programmatic standpoint, A diagnosis and fix that will ensure all text requests can be assigned to Spoke campaigns confidently the first time should become a top priority.

Happy to discuss all this more (admittedly not the easiest thing to explain here) and/or try and demo what we are seeing if that will help in any way.