For the first time the #CoSocialCa Mastodon server has started to struggle just a little bit to keep up with the flow of the Fediverse.

Michael

@mick I haven’t read the entirety of the thread, so forgive me if that’s already been covered, but have you tried defining your workers with different sequences of queues.

So you could have one service defined as

/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q pull -q default -q ingress -q push -q mailers

Another as

/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q default -q ingress -q push -q mailers -q pull

Etc.

That wat you would have 10 workers prioritising the pull queue, but picking up other queues when capacity is available. And another 10 workers prioritising the default queue, but picking up other queues (including pull) when capacity is available.

You could permutate this for some different combination of queue priorities.

Mick 🇨🇦

@michael that’s where I’m headed next I think.

I’d hoped that just increasing the number of threads for the single service would be enough, but it seems like the default queue prioritization results in a backlog and idle workers.

So dedicating a number of threads per queue seems like the next sensible step.

Thanks for the suggestion!

Michael

@mick just to be clear, what I’d suggest is not to dedicate them, but to prioritise.

Maybe you mean the same thing, but if you set up a service with

/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q pull

And another with

/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q push

Then that first process will sit idle when there is nothing in the pull queue, even if the push queue might be full.

If, on the other hand, you have a service defined as

/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q pull -q push

And another as

/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q push -q pull

Then that first command will process the push queue, after the pull queue has been emptied. And the second one will process the pull queue after the push queue has been emptied. Thus potentially wasting fewer resources.

Mick 🇨🇦

@michael right, I see the distinction, and hadn’t fully grasped what you were suggesting.

Given the limited resources available on our small server this seems like an excellent idea.

I’ll play with some changes on the staging system. Thanks!

Paul Chambers

@michael @mick

After I was told to do this, it worked like a charm and my queues haven't been backed up at all. Before, I had gotten in the rut where several thousands were backed up in queue during certain busy hours of the federation day.

Roni Laukkarinen

@mick What monitoring system is that?

Mick 🇨🇦

@rolle Prometheus, and this sidekiq-exporter https://github.com/Strech/sidekiq-prometheus-exporter

Roni Laukkarinen

@mick Thanks!

Mick 🇨🇦

@rolle I have found it extremely useful for keeping tabs on the system.

Mick 🇨🇦

@michael based on this advice I've added some services that prioritize various queues and we're humming along nicely now. Thank-you!