So here we have a six day old account with a link to an obscure promotional site in its bio, from a known bot-hosting instance, whose one and only post is an off-topic bit of Welsh political astroturfing made to an instance dedicated to anime and manga.
WTF?
It’s safe to assume it’s a bot, but it seems to be a notably stupid one. How is that even possible?
Just a heads up, if you migrated to an EU datacentre you’re potentially subject to GDPR and other legal requirements. Failure to assess that and comply could have devastating consequences.
I’d suggest the Netherlands if you need to be in that geographic area.
Thanks for the heads up! I heard that privacy laws and such are more strict in the EU so I’m reading up on what other Fediverse instances are doing about that since many of them are located in the EU.
I have only ever used hosting providers in the US before, so I wasn’t aware of the documentation requirements for many of the EU providers. Did you have to migrate the object storage as well or was that fine to just stick to where it is as long as the pict-rs configuration and urls all point to the right place?
I have never personally done a big migration like this for any of the services I run as I have usually just done small scale stuff that only I or some friends use. So, in the past I have often just destroyed things and rebuilt from scratch or, at most, just copy/pasted some docker volumes to a new host. It is something I have been thinking about a bit as I am now hosting more publicly facing things like the anime wiki, the image hosting frontend for the clips I post (they get saved to object storage), and @rikka which uses an sqlite database. I found that setting up rsnapshot to do hourly snapshots of rikka’s database has actually proven very useful for debugging issues (in addition to having a backup) because I can see the history of the database that led to the current state.
It seems like the migration is working as we made it through a Wednesday with no noticeable federation delays at all. Keep up the good work!
Thank you! I didn’t have to migrate the object storage for pictrs. It’s still in the US I believe (Backblaze put me there for some reason. I don’t remember an option to pick when I made an account.)
Yeah, it’s my first time migrating (and running) something like this. It’s almost been a year and it’s a lot of fun. What I’ve been using for backups is a cronjob and pg_dump. I heard some people use other tools like barman and backrest which I want to try. I’ll check out rsnapshot too though!
Yup. Sorry about that! I don’t think anything like that happened on the second migration (May 15) only the first (May 12) but if you noticed any, let me know!
Nyaa.social may have been a bit too on the nose (IYKYK). And as you say, Ani.social is more straightforward, so it was a good pick in the end.
No worries on the transfer, but after reading this I am wondering does this mean you plan to migrate away from EU servers once the potential fix scheduled for 1.19.5 is implemented and verified as a fix? Or will you remain on the EU servers for the forseeable future?
I’m glad you think Ani.Social was the better pick!
For migrating away or remaining on the EU, I will likely migrate back to Asia. The biggest benefit is that it’s closer to me and I think having Lemmy in other regions is in line with the decentralization idea behind the Fediverse.
The servers are very cheap in the EU though. Also, most of our traffic is from EU/US. So there might also be a chance we stick around.
I was wondering about traffic and the regional breakdown. I had been guessing it was mostly EU/NA based on the traffic I got to clips I post to the anime community using my own domain. Cloudflare just lists the top 5 countries:
However, I wasn’t sure if that might be influenced by the time of day I am posting them since I am on the East coast of the US. Also included in that image is how much bandwidth the cloudflare cache has saved me in the past month. Posting clips as webms means that cloudflare will cache them automatically and save me >90% of the bandwidth on my VPS (something I didn’t know to start with, but am grateful for).
That makes sense. I think a lot of the large instances have centralized in the EU/Finland due to world being there to avoid the exact issues that caused you to move. Also, instance <-> instance communication is probably way more requests than instance <-> user.
I was wondering what was happening, so I ended up just creating an account here. Especially since I probably spend most of my time either here or on programming.dev.
That’s what Reddthat is doing from my understanding but their project doesn’t have enough documentation for me to understand how to use it. I could move ani.social to Europe or US but I’d prefer to have the server closer to me (and to avoid centralizing all Lemmy instances in two regions).
I think the Lemmy devs are working on it but it looks like a fix isn’t coming until 19.5.
In this case, I might just move ani.social already (likely tomorrow). At least temporarily until a new fix comes.
Alright, I have been doing some poking around the grafana dashboard and noticed that about 20k activities/hour (~ 6 per second) seems to be the limit that ani.social can process coming in from lemmy.world. Whenever the activity peaks on world go over that (generally EU afternoon/NA morning), we start to lag a bit. Then, after the peak has subsided, we catch up.
All this really seems like it is putting a pretty hard limit on how big the fediverse could actually grow without federation becoming completely impossible. I was reading up on efforts that reddthat has undertaken to improve federation from world (since they are in AUS). Their EU-based proxy seems to have worked well, but even with batching like this, federation is always going to be a lot of bandwidth and message passing between servers that just might not scale past a certain point. Anyway, I am off topic.
In any case, the lag seems like it will be coming and going with a bit of regularity, kind of like fediverse tides.
The latency limit is caused by the activity queue that was introduced in v19.
Servers can only talk as fast as round time allows, because Lemmy instances now keep track that each event actually does get federated, and in the right order.
That last point means each event only gets sent once acknowledgement of the last one is received, creating a hard limit for how many events can be communicated, depending on ping. A mere two per second with a latency of 500ms.
This serial process will obviously need to be parallelized. But that’s difficult.
The federation delays seem to be getting better over the past day or so. So, whatever you are doing seems to be working. It seems like the PR to engineer out a lot of the latency sensitivity of Lemmy federation is still a WIP, so this might be something that comes and goes as the network topology and routing changes day to day. Almost like federation weather.
The outbound federation to lemmy.world has been resolved at this point (thanks hitagi!), but the inbound federation from lemmy.world continues to be an issue. I don’t really have any insight into solutions for that as it seems it might be due to physical constraints similar to what has been plaguing the AU/NZ servers.
Just a heads up to moderators of ani.social communities though. Because the world version of any community here is going to be a couple hours delayed (currently ~8 hours), it might make sense to have an account on world that you appoint as a mod to your community. This lets you take moderator actions on things that are in the world version of your community even before it federates over instead of having to wait hours for the spam/whatever to federate over before you can remove it.
Tagging @MentalEdge to let you know since you probably have some of the more active communities on the instance.
I couldn’t understand why I keep duplicating Kadath’s posts. I was positive I was checking sufficiently to be sure chapters hadnt been posted yet, but then I’d come back later and find that I’d doubled one of her posts.
Because her account is on .world.
I expect there’ll turn put to be some duplicates in the latest batch too.
So I guess when I see that something I really expect to have already been posted hasn’t been (like just a bit ago with Shiretto Sugee Koto Iteru Gal), I need to just let it go, since the odds are that it has already been posted and it’s just that .world hasn’t let us know yet…
Yeah, it’s a weird situation. I am not removing duplicates that haven’t federated over yet. Until posts federate to ani.social, no other, third-party instances see them. The situation is growing worse too. The current delay is up to ~12 hours behind.
Edit: Currently improving, back down to ~8.5 hours.
meta
Hot
This magazine is from a federated server and may be incomplete. Browse more on the original instance.