nrdufour, 11 months ago

@stefan I wonder if it's because a lot of those users are going to use the site or post/boost/etc at roughly the sam time range.

stefan, 11 months ago

@nrdufour yeah I would love to know and TBH not even the mastodon code is documented in a way that explains anything. This is all I get from the sidekiq interface:

rolle, 11 months ago

@stefan @nrdufour

Wow. Do you have multiplied sidekiq services? How much CPU/RAM? Didn't @lari have a similar issue a while back? Did you got it resolved without scale up? #MastoAdmin #Sidekiq

stefan, 11 months ago

@rolle @nrdufour @lari I'm using cloudplane for hosting and currently on medium level which is 2 CUPs and 2GB RAM and sidekiq with 6 threads.

I could jump up to the next level for twice the price to get double cpu, ram, and threads.

rolle, 10 months ago

@stefan

Ok. What I noticed that 2 CPUs and 2 GB RAM is way too little for a well federated server. I had to scale up up to 8vcpus and 32 GB RAM to keep the queues in real time sync with the world wide viral #Eurovision event.

But if it ramps up like that without a reason maybe there's a bug?

@nrdufour @lari

stefan, 10 months ago

@rolle @nrdufour @lari I'll bet there is a reason it ramps up but I just don't understand what it is.

lari, 11 months ago

@rolle @stefan @nrdufour Yes, I had something like 27k jobs waiting in the pull queue. I ended up upgrading at least temporarily to 4CPU instance with 8G memory, as otherwise it would have apparently taken days to process.

I don't know if there was some other reason for this than just growth in number of messages. There was a huge spike of Sidekiq errors a couple of days before, I don't know if it was somehow related to this.