Okay techies: Every time I post a link to my website on Mastodon, everyone (including me) is shut out of my website for about 5 mintutes with an "error 500" message.
This doesn't happen when I post anywhere else, and I get 10 time the traffic from other sources.
People have offered me and my tech support staff (my husband) solutions. None of them worked.
@Teri_Kanefield This is a known problem with the #Mastodon design which I have frequently noted, and suggested fixes for -- to no avail. When you post a link, EVERY instance that sees your post immediately runs off to fetch the link in case there's an image to display. So all those instances try to get that link essentially simultaneously, DDoSing the target site. So far, nobody in a position to do so has seemed interested in dealing with this issue.
@lauren@Teri_Kanefield This is a really good point! Some folk in the Fediverse host dynamic sites that probably would be easily drowned if something goes viral. But, I think it would not be that hard to disable pre-fetching those URLs and letting the first access by a user on an instance trigger it, thereby spacing it out over time and hopefully eliminating some unneeded access.
@tk@hoco@Teri_Kanefield Pushing the problem onto end sites, many of them very small and with essentially zero funding, who have nothing to do with Mastodon per se, is not a solution.
@lauren@Teri_Kanefield I had a related problem with my self-hosted Mastodon server being DOS'd whenever I posted an image to Mastodon. My solution for that issue was to use mod_qos on my Apache instance running Mastodon to limit the number of simultaneous requests to my Mastodon subdomain. Your problem is slightly different, but if you could find an identifying feature of the requests (e.g. Mastodon user agent?) you could possibly put in place a QOS rule to rate limit them.
@richardloxley@Teri_Kanefield I can't depend on Mastodon user agents being consistent or even being available, and I don't want to limit responses in a way that would slow down humans viewing my site (blog, etc.). The problem is a terrible design flaw in Mastodon that NEEDS TO BE FIXED. Workarounds at the web server end just gives another excuse for that not to happen.
@Teri_Kanefield shut out when following the link from Mastodon, yes, I've exoerienced this with your posts.
My Q: Ibassume the same link, pasted in the browser, does reach your website and shows the post?
(If your answer is No, that's really weird!)
@Teri_Kanefield Hard to say without seeing the logs, so definitely review the error logs for clues. Additionally consider putting it behind a CDN - there's a number of services out there for free 👍
@Teri_Kanefield I've seen this same problem when sending out an email blast — the website is immediately brought down. I suspect there is a spam bot subscribed somewhere and as soon as an email is received it slams the site with a DOS attack. Could be the same on your posts?
@Teri_Kanefield I am sorry for this, this is a known Mastodon issue. Basically, when you post a link, every Mastodon instance receiving your post will try to download the page to generate the link preview, and for popular accounts like yours this might be a few thousand of requests.
We have plans to improve this in the near future, in the meantime having some basic HTTP caching on your blog should help.
@evan Sure. This is on the 4.3 roadmap but I am not sure we will be able to deliver it, it has many implications, both in term of protocol and implementation, mostly because preview generation must be an async process.
I will try to write down our thoughts on this topic (we have been discussing this for many months), and share it to you
@Teri_Kanefield Hey Teri. Techie here. I have had the same issue on my website. The issue is (most likely) the following:
Every separate mastodon server will fetch your website so it can display it's users the link preview information. This usually means that your website receives thousands of requests (visitors) in the few seconds/minutes after you post a link. The website can't handle and goes down.
@Teri_Kanefield@dinandmentink Teri, It can easily be hundreds when you don't even have an especially well-followed Mastodon account. I can post a link to my office website here on Mastodon and then watch the raw server log scroll by and see 200 requests for the linked page come from Mastodon instances in just a couple minutes. I consider this a flaw in how Mastodon works, but it does sound like your WordPress could possibly be better tuned.
@Teri_Kanefield The moment you post a link in a toot and it gets federated, receiving instances will try to fetch your post to embed as preview on their instance. This causes a bit of overload on your Wordpress, I guess. Can be solved with some cache settings that I need to look up as I don't have them in the top of my head.
@jwildeboer@Teri_Kanefield Having your front-end web server (Apache, Nginx, …) do some request caching worked wonders for me. (I never managed to get Wordpress to serve more than a few requests per second; but since most of these requests no longer hit Wordpress, I don't care anymore.)
Also make sure the "oembed" requests get cached as well (not just to the main HTML page), as these are the ones (also) requested for the preview.
I've had problems with how link previews are generated but very different than this error 500 problem. For me the problem was stripping off the end of the URL.
EDIT: Two examples of problems with truncated links are gift links and using # to go to specific point on page (anchor links)
Anyway, if you add an image to your post then a link preview is not generated. But I don't know whether this workaround will fix this problem!
@Teri_Kanefield Interesting. I've been having similar problems when I post to my blog. I had figured it was an internal problem with my blog, but maybe it's the autoposted link on Mastodon.
Technology is weird.
Another method, is to post via a separate server instance - to serve the toot requests, leaving the primary to serve 'normal' traffic. Let the 'mirror' server grind out serving the excess traffic.
@gearhead I can't find the issue right now, but there is a githb issue in the mastodon issue list to generate the preview card for a link on the instance the link is posted from - and have the preview card distributed with the post, rather than expecting each instance to create their own card. @steveo1049@tw000@Teri_Kanefield
Add comment