Guild new message notifications not being sent intermittently

[Alys]

Three to four days ago, there were two reports in the Report a Bug guild of notifications not appearing when new messages were posted to guilds that the players were members of. The guilds' member counts were small enough that notifications should have been sent. I've copied the reports below.

The reports indicated that it had been an ongoing problem for a day or more, not just a once-off glitch. Because of this, and because at least two people were affected, and because website notifications are normally a reliable (and important) tool for keeping up with guild chat, I'm marking this as high priority, and needs investigation because we don't know what's causing this, It's intermittent so it's not going to be an obvious code bug. One of the people affected has reported that it seems to have stopped being a problem, but it will be hard to be certain of that since you'd need to check every guild you're a member of manually to see if there's new posts.

Note that this issue refers to the notifications that should appear in the website's drop-down notification icon menu. This isn't about push notifications from the mobile apps (those have known problems but anyone experiencing them should report them in-app at Menu > About > Report a Bug).

Timestamps below are in UTC.

@citrusella (2d6ef231-50b4-4a22-90e7-45eb97147a2c) 2020-05-14 07:52 :
"I for some reason am not getting notifications to my notifications list when guilds I am a member of contain new posts. It started within the past few days, maybe the past day. I first noticed it with this guild (since I regularly check it and thought it was weird there weren't new posts... but lo and behold there are!), but I've confirmed I've also not gotten notifications for the (private) contributor guild, Aspiring Legends, Aspiring Linguists (maybe?), and Aspiring Blacksmiths (also maybe, but less sure). (I think there was one other (that wasn't a contribution related or official guild) but I can't remember for sure by just comparing my still existing notifications against my total guilds list), and there are others that have a notification in my notification list that might be old--some guilds I haven't checked in a bit, so I don't know if they're also in the "notification void" for me. I'm almost kind of afraid to touch anything in case it might yield the answer as to why.) None of the guilds in question are over member threshold and some are far below it. Hmm...
"I do know it's not happening for every guild--I just got a new notification for a post to the COVID guild. And it isn't affecting mobile notifications for direct mentions."

@PixelStormArt (100fadc6-0b54-4329-8dc3-d96b5e482399) 2020-05-14 14:23 :
"I'm afraid I have to confirm that @citrusella 's issue is not a unique case. Guild message notifications seem to have stopped working properly for me as of this morning (GMT). As far as I know, I didn't have this issue last night. I had three notifications waiting for me upon logging-in (one of which was my party, I think the others were the Bulletin Board and Use Case Spolights), but upon checking a couple of my other guilds (namely Pixel Power and Aspiring Artisans: Pixels in Progress) I found there had been new messages that I was not notified about. Both these guilds are too small to have notifications turned off."

@citrusella (2d6ef231-50b4-4a22-90e7-45eb97147a2c) 2020-05-14 15:22 :
"In regards to my notifications issue, in the interest of being accurate: ...I feel the need to add that I actually am not having issues with Aspiring Legends as I said I was earlier (it was really hard to fine-toothed comb the guilds I didn't have notifications from in the list to be sure my report was complete (especially since I was supposed to be discussing dinner plans!) and I think I just included it to err on the side of caution because I wasn't sure if I'd seen the most recent post in the guild before and I knew I didn't have a notification at the time (but I do now))."

@citrusella (2d6ef231-50b4-4a22-90e7-45eb97147a2c) 2020-05-17 09:24 :
"Matteo did message me about it yesterday or so, about eight or ten hours after my own notifications issue seemed to magically clear up."

I've asked citrusella for details about what was discussed with Matteo, if it's not private. I've asked PixelStormArt if it's still obviously happening for them, although as I noted above it could be hard to tell.

_It's possible that SabreCat has been investigating this too since it was reported in RaB a few days ago.

[Alys]

Loggly has no records for either player that seem to be relevant.

[Alys]

@PixelStormArt from about half an hour ago:
"As far as I can tell, notifications are back to normal for me."

[citrusella]

So basically, I got asked if a) was I definitely talking about the notifications dropdown, b) was it happening on mobile too, and c) if it was happening on both, then could I use my browser console to look for errors

My exact response (okay, there's a couple of mostly non-relevant things cut out) was as follows:

1. The notification list next to the sync icon on the web, yes.
2. The mobile list was showing up correctly this morning, but I suddenly started getting notifications again from at least some of the problematic guilds last night on the web. I think when the problem was occurring, it happened on mobile as well.
3. Given that it magically seems to have righted itself somehow, I don't know that anything would be in the console of note (though 2 AFAICT was not true when the issue was happening). I can still check though, if you want?

Then he said it seemed intermittent and maybe the server wasn't behaving correctly at the time and I was told not to worry about the console and that it would be useful on the chance it happens again.

I said I'd keep that in mind and mentioned that it seemed weird it was happening for a few days before it cleared up, but I'd figured the sudden mysterious fix was either something on Habitica's end somehow clearing up or something in my account clearing up/changing in just the right way. awkward shrug

(I didn't directly quote things that weren't my own words here because the matter/info wasn't private but the interaction was via DM and I always feel weird quoting other people's words in a situation where I need to share DM stuff...)

[paglias]

Given the timing I suspected it was an issue linked to last week deployment but after reviewing the code and loggly I don't think so anymore. It could be due to some database or server issue but nothing obvious popped up. Let's see if we get more reports the next few days

[Alys]

A report from @anneline (16a107de-d519-43ce-8c30-8da5cbcbb237) from 4 hours ago indicating a problem in the past few days, possibly now resolved:

"I'll also add my notifications have been intermittent, and one of my party members mentioned a few days back (before the report) that her notifications had stopped. So it definitely was a thing a few days ago, and seems to be back to 'normal' now... but will keep an eye open for you!"

[shanaqui]

SuperSaraA, Friend Hugger @SuperSaraA (dbf5390d-6a57-42ad-ae14-af8fd8e570ca) [android] tier 4: Elite Socialite

I also noticed that no notifications were given for guild posts a few days ago as well. I'm sorry I don't have specifics but wanted to add that for sure I noticed something was happening and am sorry I did not mention it sooner.

[paglias]

I guess we can keep this open but it's almost certainly caused by the notifications being lost during server performance issues, they also fix themselves as soon as a new message is sent. It won't even be fixed by the use of transactions because the notifications take too long to be sent and must be handled separately from the message sending process.

Only solution would be to refactor the notifications mechanism like we discussed in Slack for guilds with > 5k members from a "push notifications to the user" to a "user pulls new notifications" system

[citrusella]

My only question is did the server really have performance issues consistently every time someone posted to Report a Bug for 2-ish days? (I use RaB for this example and not any other guild I was in with the issue because people consistently post to that one much more often than most of the other guilds I could confirm had the issue.)

Like... the fact that it was sustained (and affected some guilds completely, while fully sparing others) confuses me but I also know, like, nothing about servers or how the issues they can have work.

[paglias]

That's a good point, the server definitely didn't have issues for that long. Although I think recently we've had days with very little activity in RaB

[Alys]

There was one day a while back where we went nearly 24 hours with no posts, but the other days have had a bit of traffic. Below are the numbers of posts for each day currently visible in the guild. We delete posts we've replied to so these are just the admin replies and some posts that didn't need replies (i.e., the original bug reports would have added to these numbers if they were still there).

20 -- today
8 -- 1 days ago
9 -- 2 days ago
9 -- 3 days ago
10 -- 4 days ago
3 -- 5 days ago
6 -- 6 days ago
6 -- 7 days ago
8 -- 8 days ago
13 -- 9 days ago
8 -- 10 days ago
18 -- 11 days ago
9 -- 12 days ago
14 -- 13 days ago
12 -- 14 days ago
18 -- 15 days ago
10 -- 16 days ago
15 -- 17 days ago
4 -- 18 days ago

[citrusella]

During the time that I was having the issue, several messages had been posted before I realized, and almost every other time I checked back, there was at least one new message I hadn't been notified of. 🤔