Apparently I introduced a bug into the scheduled notes worker, so instead of creating notes it actually just created more schedules. Since those new schedules were in the past, they triggered immediately, producing more and more jobs. Like a BullMQ fork bomb, basically.
So how did that kill my notifications? Well, each time the job completes, it creates a notification. Those are stored in a redis list with a limited capacity. At full speed this thing was creating thousands of notifications per minute. This meant it would immediately wipe out the entire history and replace it with "scheduled note posted" notifications, except - and here's the thing - the note never actually posted, so every single notification referenced an invalid ID.
Then the notification service, trying to be helpful, noticed that the invalid data. It attempted to filter out all the notifications that referenced a non-existent note, which was literally all of them. All the while there's more notifications constantly arriving and immediately being filtered out.
So in effect, all my real notifications were replaced with fake ones, which were then flushed out by the system - resulting in a completely blank notifications list.
Fun!
❤️1
