Lemmy.world (and some others) were hacked

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

UpdateWhile we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

alaxitoo,

Thank you for your work 🙏

dr_scientist,
@dr_scientist@lemmy.world avatar

Good job. I don’t understand very much of that, so that makes me all the more grateful. Thank you.

TheVampireSaga,

what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?

ruud,
@ruud@lemmy.world avatar

Good point, I’ll update the post.

giant_smeeg,

Thanks! Is there any understanding as to why? Or are we thinking some script kiddies because they can?

hawkwind,

They defaced it with dicks and changed the federation list to be only threads.net. I don’t think it was a state sponsored chinese hacking group. :)

milan,
@milan@discuss.tchncs.de avatar

right after the update we also had most of the serverlist cleared except threads.net (which was the last one added so i assumed it was some bug) – otherwise nothing appears to be touched on this instance tho.

linearchaos,
@linearchaos@lemmy.world avatar

I’m ok with the dicks but the threads are TOO FAR!!! shuffles off to the angry done**

Thank you all for staying on top of it.

TheVampireSaga,

Also I am curious, what’s the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?

ruud,
@ruud@lemmy.world avatar

Mail: info@lemmy.world Mastodon: @mwadmin Matrix: matrix.to/#/-support-general:discuss.online

Container9043,

Would it be a good idea to have a secondary email not attached to lemmy.world in case of a domain hack?

ruud,
@ruud@lemmy.world avatar

info@mastodon.world

hemmes,

The mail server records of a domain name do not usually point to the same server as other services like Lemmy.

Container9043,

Domain registrar hack could happen too

sirnak,

Why wasn’t there an info on /lemmy-world.statuspage.io ?

ruud,
@ruud@lemmy.world avatar

I think the admins that were on it didn’t think of updating the status page…

InverseParallax,

Nice work on the recovery, especially from a 0-day.

BustedPancake,
@BustedPancake@lemmy.world avatar

So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I’m sure they were only interested in admin cookies, so most others were “useless” to them? I see nothing wrong with my posts so I should be safe, right?

Rooki,
@Rooki@lemmy.world avatar

If you think they could change your password:

YES, they could.

They could have changed the email => “Forgot PW” and with that you lost ur account.

00Xero00, (edited )

I think I’ve lost my account, I clicked Forgot Password and nothing came into my mailbox. This account is the one I made just now.

My old account:

https://lemmy.world/pictrs/image/4f4d816b-fd28-44b0-b760-b1084c4d3b5d.png

If you see that account post or comment on anything, please report it

Edit: Nvm, I use another email to sign up for Lemmy and forgot about it

Rooki,
@Rooki@lemmy.world avatar

Report it directly to Ruud or otherwise he will just delete it.

0Xero0,
@0Xero0@lemmy.world avatar

actually nevermind, I forgot that I use a different email for Lemmy, I can log back in now

TheSmartDude,

It happens to all of us. Additionally, assuming that you’ve come here recently, there’s not much data on it, and it being deleted will not be that much of a big deal.

Streetdog,

Same goes for my bank account.

TheVampireSaga,

Probably. I had to re-login myself.

cantevencode,

Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless

AlmightySnoo,
@AlmightySnoo@lemmy.world avatar

the details of the vulnerability are already known now anyway since there’s a fix that was proposed on the Lemmy GitHub so I don’t think it will hurt others to talk about it

namelivia,

Could you please link the issue? Thanks!

namelivia,

github.com/LemmyNet/lemmy-ui/pull/1897/files found it myself

AlmightySnoo,
@AlmightySnoo@lemmy.world avatar

yup that’s the one

what I find weird is that the “fix” still focuses only on the front-end, the issue is still that unescaped HTML is being stored in the database and still trusting the front-end is nuts

TheVampireSaga,

I think the main developers are aware of either of them but I’m not sure, haven’t seen anyone site admin wise talk about this mess.

ElectroNeutrino,

This discussion on the original bug report does talk about the back-end needing a fix as well.

Redex68,

I mean, I’m pretty sure that for an XSS attack that’s fine. The entire problem is that somebody posts e.g. a comment that contains code that is automatically run in users’ browsers. If you make the front end just not execute that code then it’s fine. Who cares what’s stored in the back end?

I mean, it would still be better to have multiple fail-safes, and they probably should still sanitize text entering the database.

But this is sufficient for a quick fix.

gabriele97,

For sure it is sufficient for a quick fix. But a Lemmy post can be posted not only on Lemmy but on other front ends (like kbin, mastodon, and many others) and they can suffer from a similar attack due to the backend storing and forwarding the bad content. So, it should not be stored as it is in the backend

TheVampireSaga,

I think people are forgetting that it’s somewhat obvious the hackers or whomever, I don’t really care honestly are Lemmy users considering they did this at night and got into the site so quickly to begin with, they’d have to have been familiar with it to get into it as fast as they did.

If anything everything should be fixed.

solrize,

Let me introduce you to my friend, Little Bobby Tables… :)

luthis,

ALWAYS SANITISE!

sv1sjp,
@sv1sjp@lemmy.world avatar

Oofof… That’s not suppose how we announce vulnerabilities…

MeshPotato,

Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I’m genuinely hopeful that this will be a good alternative.

LuckyLu,

Very impressed by how quickly action has been taken by this and other instances to patch the issue.

grissee,

uh, why did you have negative one dislike?

LuckyLu,

Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.

grissee,

huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances) https://lemm.ee/pictrs/image/54a81fea-44a5-43ca-8689-8ea612f5612c.webp

LuckyLu,

Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).

Not exactly sure I understood what you meant by “either positive or zero”.

grissee,

see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)

but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior

for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7 https://lemm.ee/pictrs/image/7c412caa-6c22-485d-9184-ebe7692486f0.webp

LuckyLu,

Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.

That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.

Darkard, (edited )

Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.

I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.

In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

LuckyLu,

I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD

No1ButtMe,

I was I able to upvote anything or subscribe. Seems like uninstalling and reinstalling fixed my issue

Infernal_pizza,
@Infernal_pizza@lemmy.world avatar

I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it

LazaroFilm,
@LazaroFilm@lemmy.world avatar

For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.

calaei,

Go into account settings, clear your password, re-enter your password, save, go to feed and pull to refresh. That’s what worked for me.

scarabic,

No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.

I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”

Carnelian,

So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear

What I did was add my account (again)

There was no warning or anything, and it populated the list with two of me.

At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?

I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.

I logged back in, and now everything is back to normal

LuckyLu,

Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

LuckyLu,

Whoops, glitched double response.

JJROKCZ,

Finally I found good instructions, was about to delete and reinstall until I followed this!

LuckyLu,

Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

Pandantic,
@Pandantic@lemmy.world avatar

I did this, but I just didn’t delete either accounts and it worked fine. Idk if it’s detrimental to have two of the same but it worked for me.

nan,

I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).

scarabic,

Ah interesting. I’ve had multiple accounts from the start so it was much easier for me. Just removed my main account and added it back.

deweydecibel,

In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

Good PSA. It took me a bit to figure it out, the app doesn’t make this obvious.

ellaella_ayayay,

Oh, I was wondering why it was showing me as logged in but wouldn’t let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!

Caboose12000,

thanks for posting this, I wouldn’t have figured that out lol

nei7jc,
@nei7jc@lemmy.world avatar

How have I never thought of comment hijacking?!

trouser_mouse,
@trouser_mouse@lemmy.world avatar

Very, seems like great work.

giant_smeeg,

Was the vulnerability known before hand and not applied to this instance or is it new?

ruud,
@ruud@lemmy.world avatar

It’s not fixed yet in the current version

AlmightySnoo,
@AlmightySnoo@lemmy.world avatar

see the GitHub repo, it’s new

hawkwind,

Concerns were posted a few days ago, but no POC that used the exact same attack as we saw here. Basically, there were some warnings, and work was underway that would have prevented this, but it was not done fast enough. There is a patch now, that will take a while to roll out, plus a renewed focus on general and related issues.

JohnSaveourSocks,

Rock on, Rudd.

nosut,

Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

EDIT: it has been added back to the block list.

Carnelian,

Hey how do you check on that?

As of the time of me posting this comment, exploding heads is appearing in my feed with some anti lgbt posts. Idk what’s going on because I’m pretty sure they’re supposed to be defederated currently

Reliant1087,

Thank you for taking the time to update this :) Hope everything will be sorted out without people being scared. As a layman, was any user data compromised?

brittleback,

Well done all involved. Sounds like it was caught and mitigated quickly

bluemellophone,

Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

ruud,
@ruud@lemmy.world avatar

Good point. I did post about this on Mastodon @mwadmin

bluemellophone,

Thanks for the reply!

hawkwind,

Don’t fall for it. They’re also an admin on mastodon.world! :)

FlyingSquid,
@FlyingSquid@lemmy.world avatar

Now I don’t know who to believe! Is Lemmy even real?

danielton,

Are we all just lemmings? Oh…

JohnDolt,
@JohnDolt@lemmy.world avatar

Good shit! Thanks for keeping things up and the pretty quick response as well.

cantevencode,

Does an admin account have any permissions to view email addresses or data of registered users?

Did MichelleG not have 2FA enabled?

Now that this has happened, it’s be worth pushing this issue through as high priority. If HttpOnly was enabled, then an admin takeover would not have been possible.

github.com/LemmyNet/lemmy-ui/issues/1252

nosut,

The JWT exploit bypasses 2FA requirements. It basically steals your active session and allows a third party to use it.

cantevencode,

Good point. I suppose the only way to fix that particular issue to disallow cookie authentications from a new location

sudneo,

Using proper cookie flags can also mitigate this. I am not sure there is a reason to have the session cookie accessible via JS. HttpOnly flag alone could have helped here.

V4uban,

Thank you for your fast answer!

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • lemmyworld@lemmy.world
  • DreamBathrooms
  • InstantRegret
  • thenastyranch
  • magazineikmin
  • GTA5RPClips
  • rosin
  • osvaldo12
  • tacticalgear
  • Youngstown
  • slotface
  • khanakhh
  • kavyap
  • ngwrru68w68
  • modclub
  • Leos
  • everett
  • provamag3
  • cubers
  • cisconetworking
  • ethstaker
  • Durango
  • mdbf
  • anitta
  • megavids
  • normalnudes
  • tester
  • JUstTest
  • lostlight
  • All magazines
    •