[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

MrPear,

Thank you for the transparancy!

GatoB,

I love post like this, another reason of why this instance is so good

DRx,
@DRx@lemmy.world avatar

Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.

Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though

any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?

morelikepinniped,

Thanks for all your hard work @ruud. It's not easy!

slimerancher,
@slimerancher@lemmy.world avatar

Oh. Good luck with the next try!

Spitz,

Jerboa crashes constantly for me now, doesn't even let me view anything. JSYK!

Fog,
@Fog@lemmy.world avatar

There's a fix the dev has just released. See reviews section on the app. He said it'll take a few hours to show in the playstore.

ThisIsNecessary,

This is good news! I haven't been able to try Jerboa fully yet since I use lemmy.world. I'm not seeing this review section you're talking about but I'll be on the lookout for an update. Thanks for sharing!

Fog,
@Fog@lemmy.world avatar

Sorry, my comment wasn't very clear. On the app page within the playstore, if you scroll to the bottom and read the reviews you'll see the developer has replied to say a fix will be released today. I'm yet to see it though.

Gamera8ID,
@Gamera8ID@lemmy.world avatar

You might want to try https://wefwef.app

snargledorf,

@ruud just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

neblem,

Is there a way to have two instances writing to the same db? That'd help a lot in this situation so the test instance can still be accessing the same data as the stable instance. Otherwise we'll never have enough load to fully stress test and be in the same situation as the three existing Lemmy test instances.

snargledorf,

Seeing that with these updates often come with changes to the database as well, that could be very difficult if not impossible.

ruud,
@ruud@lemmy.world avatar

Yes, considering that. But we'll need people to use that when we will do testing...

snargledorf,

Well, you have at least one person who would be willing to use it ;D

med,

One more. You find a stable way to notify anout upgrades and get a test sheet to run through and we can generate posts and activity to help test with.

Light the beacons! lemmy.world calls for aid!

Finnagain,

I'm not familiar with what the server architecture looks like, but is there a possibility of using a load balancer in front of the instance's server and swapping a "beta" server into the load balancer when you need to do testing? You could basically migrate your traffic with zero downtime, assuming Lemmy's architecture allows for it.

ruud,
@ruud@lemmy.world avatar

Well that doesn't differ much from what I do. I just copy the files to a second directory and test with that. Easy rollback. Downside is, that all data is lost between upgrade and rollback, which will be the same in the scenario you suggest.

Finnagain,

Ah, that's fair. Best of luck either way. This is the tough part of admin work. FWIW, the stability we've had thus far has been pretty impressive!

noXi,

was there a test environment? why rc in production?

HangoverTuesday,

We are the test environment. You get what you pay for.

forksandspoons,

Linking from else where in the post... Theyvdid push into a test environment, but the issue here resulted from the immense user load in production which did show up in the test environment.

https://lemmy.world/comment/612890

snargledorf, (edited )

RC2 RC4 was just released, here's hoping it solves the issues you were experiencing with performance of RC1!

rcmaehl,
@rcmaehl@lemmy.world avatar

I thought RC3 was just released?

snargledorf,

Dang! Their moving fast!

I see RC1, RC2 and RC4!

T156,

I was about to make a joke about RC4, but no, it's a real release now.

FollyDolly,
@FollyDolly@lemmy.world avatar

After the update I can no longer log in or interact with Jerboa. Hopefully y'all resolve this soon, as I am really digging Jerboa, but thanks for letting us know what's going on, and keeping up the good work.

Spliffman1,

I uninstalled, then installed the fork of Jerboa a guy did from the post in jerboa community, has compatibility with old 0.17 lemmy, I signed in fine, dismissed the warning and all is good, no crashing

T156,

There's a major breaking change in the 0.18 update (moving away from Websockets for posts and comments), that means that older Jerboa builds no longer work with instances on, or past, that version, and newer Jerboa builds won't work with servers older than 0.18.

Dran_Arcana,

Version fracturing is probably going to be a fact of life on a decentralized platform like lemmy, a "good" app should probably handle cases for the latest few revisions instead of just the latest.

T156,

It normally would, it's just that 0.18 is a breaking change, incorporating a major rewrite of how posts and comments work on Lemmy.

nerdyshades,

I really like Semantic versioning. This would be from 0.17.X to 1.0.0 as it's a breaking change to the public API.

https://semver.org/

neblem,

Semver pre version 1 lets minor changes be breaking (points 4 & 5).

JonnyJ,

I'm sure it will eventually, but early stage tech needs some allowances while they handle growth and scale. Better to break stuff now than be hamstrung by tech debt years later

jetsetdorito,

I've been using Jerboa since I joined a few weeks ago, I haven't been liking any of the other clients so far, but today I tried thunder and I'm enjoying it more than Jerboa

caut_R,

Bummer. What keeps me from using Lemmy more atm is that it‘s unresponsive and gets stuck a lot on my end (iOS). I was hoping the update would improve on this. I hope future upgrades work better. Thanks for your work!

MarsAgainstVenus,
@MarsAgainstVenus@lemmy.world avatar

It seems that my Hot feed has been stuck for the last day or two.

T156,

Looks like we're still having issues, more's the pity.

Do we know what's causing the issue, or is it still a bit of a mystery for the time being?

heffa,

Maybe next time GGs

rrobin,

As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.

We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.

T156,

Although since we seem to be rolling more than a ship in a storm, I think a proportion of lemmings would end up hospitalised for alcohol poisoning.

oceane,

Who talked about alcohol?

Jones885,

That's alright. Thanks for all your hard work!

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • lemmyworld@lemmy.world
  • GTA5RPClips
  • DreamBathrooms
  • thenastyranch
  • magazineikmin
  • Durango
  • cubers
  • Youngstown
  • mdbf
  • slotface
  • rosin
  • ngwrru68w68
  • kavyap
  • tacticalgear
  • ethstaker
  • JUstTest
  • InstantRegret
  • Leos
  • normalnudes
  • everett
  • khanakhh
  • osvaldo12
  • cisconetworking
  • modclub
  • anitta
  • tester
  • megavids
  • provamag3
  • lostlight
  • All magazines
    •