UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.
Original post follows:
This post is intended as a central place that admins can reference regarding the XSS incident from this morning.
What happened?
A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.
Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.
Am I vulnerable?
If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.
I had custom emojis on my instance, what should I do?
This should be enough to mitigate now:
- Remove custom emoji
DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;
- Rotate your JWT secret (invalidates all current login sessions)
-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();
- Restart Lemmy server
If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)
Legal
If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402
How did the hackers get the cookies in the first place? Compromised devices on the clients?
I’m not affiliated with lemmy.world or any other instance but I do software and can explain some of the jargon above. XSS is the abbreviation of one of the most common attacks we see on websites, cross-site scripting. This attack works by having some vulnerable code which arbitrarily executes some javascript on a users browser.
In this case, the attacker seems to have found a vulnerability where a specially crafted character is executed when users read posts or comments containing it. In this case, it was especially bad because of how passwords are stored. When you log in to pretty much any website, passwords are stored in the form of cookies; small pieces of data that are passed back and forth to the web server and the client automatically. Usually, these cookies are set to not be readable by javascript, but in this case, it appears that that flag was not set. This allowed the XSS exploit to be sent back to a computer which was set up to grab these cookies.
One thing to note is that although the cookies were stolen, our passwords wouldn’t have immediately been compromised. Our login cookies store jwt tokens; a cryptographically signed message proving that we provided a password previously. That’s not to say that further escalation isn’t possible, but its hard to say for certain how far the hack went before being noticed.
Thank you for the detailed explanation :)
Stealing sessions/cookies is the new trend, it bypasses 2FA and hackers do not even have to know passwords.
Client does not have to be infected, it can happen inside the browser, there is no known protection at this time.
You can only mitigate, log out after you finish the session, like banking and clean or do not save essential cookies.
There are various vectors though: https://blog.netwrix.com/2022/11/29/bypassing-mfa-with-pass-the-cookie-attack