Is there something wrong with the connection between #qoto and pixelfed.social? I follow ceoln@pixelfed.social from ceoln@qoto.org, but when I try to look at it, the latest image (posted on April 3rd) is missing, and none of the actual images appear (just their alt texts). Or have I just forgotten how things work? :)
Here's an example where, when I click on it to find the context, I don't see the preceding toot, even though one of the people mention is a mutual follower with me.
@dannotdaniel Ya, sounds like your instance admin is filtering stuff before it gets to you. I've got the same problem at #qoto as it happens, but it's even worse than that at Outsourced Math (Friendica) where all images are filtered, requiring you to click on 'em to even see if you wanna click on 'em. They weren't always like that, but it's a horror show now.
Hey @freemo , is it that an RSS-to-mastodon service at https://feedsin.space/ is blocked on #qoto? I've been using this a lot from my Bulgarian-language account, and I just noticed I don't have access from here.
I find such services very useful. If there is an intentional block in place, could you share the reasons and if you could suggest alternative ways to follow RSS feeds from here? Just trying to understand if there's a solution that would work for everyone. Thanks!
My attempt to migrate here from @hesgen is not going to plan. Moving followers and redirecting from qoto.org is either broken or not implemented in its 'fork' of Mastodon.
Seems our load balancer is a little under powered and its stalling momentarily from time to time. Most of you probably didnt even notice it but some of you may have.
I am going to upgrade that today to the next level. May result in up to 5 minutes down time (the new architecture will have multiple load balancers).
Looks like #QOTO is now stable after three weeks of upgrades. The #DailyBloggingChallenge have been somewhat out of whack for the month of February, thus to put all back in sync, will be posting twice a day for the next weeks.
Either some of the fixes I did yesterday wound up addressing the latency problem we had, or it was intermitten. But everything has been operating fine for the last 12 hours or so. I will continue to monitor the situation but for now everything is in the green on #QOTO
Assuming no more problems trigger we and the team are continuing on building our beta environment which will enable for smooth update role outs int he near future. We have about 5 updates we developed in house we want to apply in the near future.
@freemo Is there any kind of “status” page for qoto.org that is independent of the mastodon instance? Or somewhere that lists known issues?
I’m experiencing significantly degraded performance and I feel like it would be good to have a place to go to find out if this is scheduled maintenance, a DDoS attack, or something else.
@pganssle Yes I update people on #QOTO but its not independent of the server. I do however reboost it from my ACM account which you can follow to see that .
So while that last fix did make things better, there seems to be a bit of lingering slowness for a few things, but things are mostly working.
I worked all day(and its midnight now) to bring up some very extensive monitoring tools for the team for us to help diagnose the issue.
I will work through the weekend to help improve things further, though I may need to get some sleep before I can fully resolve this. At least things are working other than a slight bit of lag. I will keep everyone on #QOTO updated.
Today we had some problems on the server and it was slower than usual. Had to do with a mismatch in versions on docker engine. I managed to keep the server running while i diagnosed the problem, but it was noticably slow with the occasional 404.
Now that i found the problem I am applying it but you should already notice things are mostly back to normal. In a few minutes everything should be fully responsive again.
Sorry to everyone on #QOTO for the recent difficulties, this was a very big migration for us and we are doing a LOT of work.. so breaks a few things up front but in the long term it will mean more updates, faster updates,and a more stable scalable system.
We still have about 5 updates we are planning to apply soon, we are just perfecting the environment first and setting up a beta environment, so stay tuned.
(I saw this because i looked at an account i made somewhere else to bridge the time until qoto was back online)
I tried to find the first line in my home list, scrolling down to 8h in the past with the boost only 2h old. Also tried eliminating all special characters, "ihr lieben" alone does not show up either.
I also checked that i can see the original toot author in the qoto web interface, no problem there.
One thing i could think of is that maybe, if the original toot is older than our recent update, it will not show up anymore? If that was the only problem it would be fine, but i am a bit worried it might not.
Everything at #QOTO is now fixed and the all queues are response again. Sorry for todays slowness, but we found the last of the problems from the migration and now fixed that as well.
after another short down time everything is fixed! #QOTO had to recreate an index that got lost in the migration which was slowing down the DB and everything else. No amount of resources was going to help.
But it is fixed now all ques are empty or very close to it. we will now downgrade the DB to a more sane level now that it is fixed (we upgrades shortly to maintain the system). But it will sill be a pretty hefty system for us, and we can always scale back up when needed.
TL;DR everything is wording fine now.
PS we are now going to work on a staging environment to test upgrades so we can safely start moving the main server through the upgrade cycle. stay tuned.
I just upgraded the #QOTO DB server to x2 the CPU. This seems to fix the underlying issue. The pull queue (not related to most things) is now recovering as well.