What kind of resources does your server have, how many users do you have, how many followers, how many are followed?
I can tell you how to run an instance with thousands of users, using loadbalancers and spread out sidekiq processes, but that might be bad advise if you just need a bit more uhmpf in the right place.