My apologies for the recent spate of problems. I think I’ve narrowed the problem down to the /m/fediverse and /m/random magazines. For some reason, mbin is generating an enormous amount of outbound delivery messages for these two magazines. I first tried removing the hashtags from /m/fedivese, but that was only a quick fix. So I deleted the magazine. (Note, the notifications appear to be related to the “microblog” function, and were originating from accounts on lots of mastodon instances, so I think there is a bug somewhere).
I noticed /m/random doing something similar. I have removed all the subscribers from that magazine to try to reduce the number of notifications it is sending. I don’t know if that will help - I have a feeling the instance can’t keep up with that happening in both random and fediverse.
Anyhow, the queues are draining fast now. I purged about 600000 queued delivery messages that (based on a random sample) all appeared to be associated with fediverse and random. That should let the rest of whatever is backed up get moving again. and hopefully stay moving.
Btw, Fedia.io is running on a latest gen AMD Epyc with 96 CPUs, an nvme ssd, and 256gb of ddr5 exc memory. And Fedia is the only thing running. So it’s not a capacity issue, but rather some challenges with the code we keep bumping into because (I think) we are pushing it harder than other instances.