Few people asking what my server resources look like during one of these events. Some graphs attached. For reference, the post was created on Mastodon at 15:31.
@hugopoi Network usage in terms of what? Like throughput?
It’s a Plesk server running Apache and Nginx as a proxy. I’m using Kirby as my CMS, so it’s Linux (Ubuntu 22.04) and PHP (8.3). Kirby is flat-file, so no DB involved. It’s all cached by Kirby too.
Those graphs look fine to be honest, and if you're serving static HTML then all PHP performance comments can be voided out. My best guess now would be that you're not currently serving the statics with nginx, and/or not currently using gzip compression.
If you're comfortable DMing/emailing over a Plesk login I'd be happy to take a proper look for you. Obviously totally understandable if you're not!
@kev varnish or another caching solution in front of your site generator might help. Note, this is local, not like cloudflare that I agree, would be icky.
@signaleleven@kev Yeah, we use WP Fastest Cache (free version) and it works really well; most page requests to our site don't even hit php, they're served from a cache by nginx. But that also falls into the sysadmin realm, as it requires configuring things on the nginx side.
@kev is this a dynamically generated blog page from Ghost or WordPress or something? Dynamic generation may cause a performance hit if you're using that. My site serves completely static pages from nginx and uses a similarly specced VM and can (and has) taken the full hit of frontpaging HN without CDN caching enabled without skipping a beat.
@kev one of the most impactful things you can do is lowering the bar for what the server has to do to serve a page, ideally the most it should have to do is read from disk or a memory cache and serve it. Talking to a database and having to generate HTML on the fly for every request is going to nuke your performance pretty hard, and unfortunately that's how a lot of blog software rolls. Hugo and Jekyll are my preferred blog generators right now because your run it once and it spits out HTML files
@kev@alyx How is the "want more content?" bit working? That appears to change on every refresh, and since I don't see a separate request for it that indicates some kind of dynamic generation is happening for the HTML. Even if all you're doing is gluing static files together and returning them, that still takes time.
The fact that you aren't CPU bound with this jumps out at me, too. If nginx is just returning static files you should be able to go up to the limit of your network interface.
@ThirstyGayGeek@alyx Great point, I completely forgot about that little box. It’s a snippet of php that takes my post titles, randomises them, then picks the first 3.
I’m gonna do some testing with that box removed. Thanks.
@kev Wouldn't it be neat if we could stop thinking we need social/preview cards? The servers could just leave it to people to click the links themselves at their convenience, like in the olden times. (I'm mostly serious)
@getkirby I’ve tried staticache in the past, but it gave me problems (can’t remember what they were). I’ve just re-enabled it and I don’t appear to be having any problems now, so I’ll test it next time I post a link here.
@getkirby I have a few more things I wanna test, like Varnish and a couple other cache solutions, then I’ll likely write a follow up post about it, which (hopefully) won’t bring my site down. 😂
@kev it's only looking for meta info, so you could try slapping the entire HTML body inside an if(!stristr($_SERVER['HTTP_USER_AGENT'], 'Mastodon'){} so that it only parses if not a Mastodon server.
Also, if you're running PHP as an FPM app, you can control how many instances it fires up in parallel. Be careful not to allow too many at once as memory hungry scripts will eat your resources, but processing more of the queue at once might help. PM.Max_Children setting in Plesk's PHP config page 👍
@kev note that the point isn't that you need to use cloudflare, just that there is a specific endpoint you and put some caching on. Do this however you can to reduce or eliminate the problem
@kev Would it be better instead if the "card" was generated and fetched by the client instead ? There are much more clients than instance out there, but people won't read your post at the same time, and their mastodon client could decide or not to show these cards
@JoeRess Nope, Kirby (still PHP based, but a lot lighter than WP). Nginx sits in front of apache for cache and there’s a load of resources free on the server.
@JoeRess fingers crossed. I was hoping a CDN for the image would sort it, as that’s the lion’s share of the data that these requests pull, but it seems it’s the number of requests, rather than the size in KB. :(
@JoeRess@kev i run my wordpress blog (along a bunch of other stuff) on a $5 vps and i've never had this issue... guess it depends on your followers and/or the virality of your post
@valerauko@JoeRess It’s based on followe numbers. I never had this problem when I had a few hundred followers, or even a couple thousand, but now I’m at ~26,000 it’s a problem.
I’m not sure at what point that pendulum swings, but it’s certainly a problem for users with more followers. I don’t know how many followers you have, so can’t comment on that.
@kev when you say "nginx sits in front of apache for cache" do you know for a fact that it caches?l and doesn't propagate all requests to apache (and php)? It's unusual to use nginx for that, I don't even know how (though I think it's possible)
Ideally for static content (including dynamic pages that can be cached for a while) you should not even hit php in the backend. Hence all the people shouting "varnish".
@kev oh ok... It says that 😅
If you can see the apache logs, the test would be querying the same page several times and seeing that apache only logs it once. If that's the case, I'm really underestimating the volume of traffic...
@shellsharks I imagine static sites can probably handle the load better as they’re usually hosted entirely on a CDN. So I imagine that pain is far less.
@kev I read about this feature the other day. What could be done to mitigate this effect? Maybe sharing link previews with other servers rather than having them generated each time, but I couldn't say how much work that would entail - or how long it would take to get the other Fediverse servers to implement that kind of change.
@kev@lehi sorry, didn’t mean to disrespect. Just read about the issue from here and it suggests that there might be something wrong with your server if it goes down under this kind of load.
@aqunt@lehi Boosts won’t affect it though. It’s when the link is first posted that’s the problem. When someone boosts it, unless the person boosting has a tonne of followers on instances that haven’t already “seen” the link, it won’t make a difference.
@lehi The thought I had was that the instance to which the link was posted be the “canonical” source. That pulls the meta data, then other instances pull from that instance’s cache.
I have no idea how much refactoring of code that would involve though. I imagine quite a lot.
Add comment