Over the last few days I'm getting a lot of errors in my Mastodon UI.
In my web server logs I get this error a lot:
ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use)
Still not been a me to fix this. Here is what I’ve tried:
Played with the max_client_conn and default_pool_size settings in pgbouncer
Played with DB_POOL in .env.production
Turn off pgbouncer
Restated pgbouncer
Restarted all mastodon processes.
I am out of ideas
(Single user instance running on a fairly low-spec machine running glitch-soc, by the way, but has been running stable until the weekend (when I last merged in from glitch-soc/main)
I’m currently restarting mastodon-web every 10 min which sort of just about keeps it alive but is clearly not a solution…
@michael Have you tried with Vanilla Mastodon? Does it work with that? That would be my first try. Usually errors happen when using some code or branch that has a mistake. I have no clue about glitch-soc but there would be my first bet.
I myself have been having to revert back to a commit or even do a hard reset to vanilla couple of times because of a faulty merge or some literal glitch in my changes. Sometimes you just have to start over or go back a bit, if nothing else fixes it.
It looks like I finally managed to resolve this by increasing postgres’ max_connections setting, together with introducing the DB_POOL setting in my Mastodon enviroment:
@michael@mwadmin
Increasing max_connections does indeed increase the amount of shared memory used, but it doesn’t come out of shared_buffers, so that advice seems ill-informed to me.
I am not saying you shouldn’t do it, I am just saying that the justification is wrong.
Not an expert but having read about such postresql issues, the answer always seems to be that you need to set up connection pooling. It seems you hit the limit for the number of allowed simultaneous postgresql connections.
Are you using connection pooling? From what I read, this limit you hit is an easy limit to hit without pooling. And, of course, postgresql does not support connection pooling natively. You need to add it.
There are those much more expert than me, but here's my thoughts. Ignore if you are already using connection pooling.
Have you installed pg_bouncer? It will keep a number of connections open to the database (saves time) and lets them be shared by a number of processes simultaneously so you won't run out of connections.
There are articles that help in the non-straightforward calculation of the optimal number of shared connections.
Add comment