supakaity

supakaity, 8 months ago

Migration has been completed!

supakaity, 9 months ago

You are super welcome, lovely.

It brings Ada and I a whole heap of pleasure running these instances and it’s largely knowing that we’re making a difference to our users, and providing a safe space for you all to grow and flourish that makes it all worth it for us.

blobhaj, hug, tinybla

supakaity, 9 months ago (edited 9 months ago)

The maintenance has completed and everything is now running from our new database.

Thanks for your patience while we performed this migration.

blobcat, code

blobhaj, heart

supakaity, 9 months ago

I have been watching my love tie herself in knots over the last several days, having to deal with the drama that has been brought on, trying her best to bring everyone back together.

There’s been bad behaviour from both sides, and I’m really disappointed to see that some of the worst of it came from our users, who didn’t keep to the moral high ground, disregarded our instance rules and stoopped to levels of behaviour worse than that leveled against them.

There have been accusations against us (or Ada specifically) that we are a safe harbour for bad behaviour and cause harm to trans people through our inaction.

This is perhaps the cruelest accusation they could have leveled at Ada, as she works tirelessly to maintain a safe space for our community, and while I was hoping, for all the effort that she was investing into this issue, that she could make it work despite my own reservations, this last attack on her impeccable morality has made me very angry.

I’m sorry for those that wanted to remain federated, sorry that it came to this, but I am glad it’s over now, purely for the mental health of my precious beloved.

supakaity, 10 months ago

So, one thing I’d mention is the systems and admin work involved in running an instance.

This is on top of the community moderation, and involves networking with other instance admins, maintaining good relations, deciding who to defeferate from, dealing with unhappy users, etc.

Then there’s the setup and maintenance of the servers, security, hacks, DDoSing, backups, redundancy, monitoring, downtime, diagnosis, fixing performance issues, patching, coding, upgrades etc.

I wouldn’t be here doing this without @ada. We make a formidable team, and without any self effacement, we are both at the top of our respective roles with decades of experience.

Big communities also magnify the amount of work involved. We’re almost at the point where we are starting to consider getting additional people involved.

Moreover we’re both here for the long haul, with the willingness and ability to personally cover the shortfall in hosting costs.

I’m not trying to convince you to stay here. But in addition to free hardware, you’re going to need a small staff to do these things for you, so my advice is to work out if you have reliable AND trustworthy people (because these people will have access to confidential user data) who are committed to do this work long term with you. Where will you be in 3 years, 5, 10?

supakaity, 10 months ago

To be clear, $3k is an accurate, but unacceptable amount.

As in that’s what it’s actually costing us, but it’s not what it should be costing. I’d imagine more like $250 is what we should be paying if I wasn’t using AWS in the silly way I am.

I’m admitting up front that I’ve been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.

Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.

And so I didn’t notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that’s what got me.

About half that $3k is just data transfer costs.

Anyhow the notice was just to let our users know what is going on and that there’ll be some maintenance windows in their future so it doesn’t surprise anyone.

We have a plan and it will all work out.

Don’t panic or have any kneejerk reactions, it’s just an FYI.

supakaity, 11 months ago

Migration complete.

supakaity, 11 months ago

It wasn’t an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately publicUrl is not a valid URL, so lemmy’s /api/v3/site metadata endpoint started returning an error relative URL without a base instead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.

supakaity, 11 months ago

How do you know I haven’t always been the hacker who’s in control? :D

supakaity, 11 months ago

Okay, I'm taking down the server now to upsize thew instance hardware.

supakaity, 11 months ago

All fixed!