derek, 19 days ago

@jan @michael was this the same issue you were seeing with your sidekiq errors?

jan, 13 days ago

@derek @michael
@derek @haploc

So it seems it's possibly a bug, but the odd thing is that it doesn't hit everyone.

https://github.com/mastodon/mastodon/issues/18063

https://github.com/mastodon/mastodon/pull/27138

#mastoadmin

michael, 13 days ago

@jan @derek @haploc

I’ve been restarting my sidekiq on a daily basis pretty much since day 1 anyway, because there used to be a memory leak somewhere (I have no idea whether that’s still the case, but never bothered to retest), and yet appear to have been affected by this.

However it has to my knowledge only happened once to me, during a period of network connectivity problems, so maybe that’s related? Very hard to tell, of course

paul, 13 days ago

@jan @derek @michael @haploc

Do you happen to use Backblaze B2B for your S3?

This is a known problem with Backblaze B2B users, esp when media cleanup tasks are running.

I've had an open ticket with them for months going back and forth to get it resolved.

These errors all occur around the same time: "Failed to open TCP connection 443 (execution expired)" errors, "No Tomes Available", "end of files."

They have elevated to senior support, we've replicated in testing, etc.

paul, 13 days ago

@jan @derek @michael @haploc

While running 'tootctl media remove' with them monitoring the last time, I got this many errors:

157 "Failed to open TCP connection 443 (execution expired)" errors, 394 "No Tomes Available", 105 "end of files"

It ran for around an hour. I forget how many files were removed. Not sure all of it was related. These were the errors that came out of the logs during the test.

jan, 13 days ago

@paul @derek @michael @haploc

Interesting. I do use B2 S3, and indeed, the issue only started after migrating to that.

I'm wondering why I'm having a lot of open/pending connections to Elasticsearch, though, if it's related to B2?

michael, 13 days ago

@jan @paul @derek @haploc

Yes, I also am using B2. No idea if there’s a connection. Could be coincidence too, I suppose.

paul, 13 days ago

@jan @derek @michael @haploc

I meant to add, I get the nofiles too. I've raised the limit, setup sidekiq restart crons jobs, (I think by the advise of Michael) and it still happens, a lot. That was the point of my initial reply and should have been the start of my initial reply before the backblaze info.. ;)

mick, 8 days ago

@jan @paul @derek @michael @haploc there’s a bug in the ES handler, it never closes sockets.

mick, 8 days ago

@jan @paul @derek @michael @haploc there’s a patch for this, I’ll dig it up when I’m back at keyboard.

mick, 7 days ago

@jan @paul @derek @michael @haploc https://github.com/mastodon/mastodon/pull/27138

I think most admins restart Sidekiq regularly and/or don’t connect to Elasticsearch over https so they don’t encounter this one.

We’ve been running with this patch since October.

michael, 7 days ago

@mick
Thanks for this. Very helpful!
@jan @paul @derek @haploc

jan, 4 days ago

@mick @paul @derek @michael @haploc

Applied the patch, let's see what happens.

paul, 2 days ago

@jan @mick @derek @michael @haploc

To apply this patch you would change this file, (config/initializers/chewy.rb)

https://github.com/mastodon/mastodon/pull/27138/files

then RAILS_ENV=production bundle exec rails assets:precompile
then restart Mastodon?

jan, 2 days ago

@paul @mick @derek @michael @haploc recompile was not needed.

paul, 2 days ago

@jan @mick @derek @michael @haploc

Thanks.

mick, 2 days ago

@paul @jan @derek @michael @haploc unclear to me that the asset precompilation step is necessary here, but I’m not the most accomplished Rails admin, so I’ll defer on this.

That’s the correct file.

And ya, restart sidekiq, specifically.

paul, 2 days ago

@mick @jan @derek @michael @haploc

Great. Thanks Mick for following up with the patch.