Lemmy.world has been down between 02:00 UTC and 05:45 UTC. This was caused by the database spiking to 100% cpu (all 32 cores/64 threads!) due to inefficient queries been fired to the db very often.
I’ve collected the logs and we’ll be checking how to prevent this. (And what caused this)
lemmy.world is a test environment for Lemmy developers… 😅 Jokes aside, issue is an issue.
Well, as pretty much the biggest instance, it provides the best data for load-testing. 🫣
All Lemmy instances are test environments right now. It’s just that lemmy.world is being tested the hardest.
Every Lemmy update:
“We fixed some performance issues by optimising some queries.”
Also: “To balance it out, we added some new even more inefficient queries.”
Next time just god damn upload it directly. Thx
What do you mean?
what You can just simply copy the video directly and paste it directly into the comments
I don’t know how to do that from Android and into the Summit app.
There should be an icon like this:
And you can probably upload a picture there
There is, but my file was a video, maybe that’s why I could not.
(And what caused this)
Prediction: bad database programming. ;)
Are we extracting enough value out of our volunteer developers and DBAs?!
The beatings will continue until morale improves!
Or clever denial of service attacks
Thanks for keeping us updated. FYI i noticed an issue there was an error message saying to check (the matix) and (somewhere else, Lemmy community support?). Both of them pointed to the same URL, but im sure they were meant to point to different places.
Edit. Happed again and I took notes. Both point to lemmy.ml community support
It seems to start at roughly the same hour every day, at around 01:20 UTC
Yup. We might be on to something now
Another DDOS attack?
People move to smaller instances so that with such outage not everyone is affected. Use fediverse as its supposed to be used.
What’s the name of the server you are running?
A large instance today will be a small instance in the future. There are hardly any users on lemmy compared to other more established platforms. So if lemmy is to ever handle a lot more users, stress testing the code makes a lot of sense.
What’s going to happen in the future, do you expect there to be 50,000 servers? That’s unrealistic.
You’re not taking into account that some people are dumb as fuck. They will sit on one instance and when the instance goes down , they’ll start whining
I can’t claim to know what the designers intended, but having users spread across a large numbers of servers is terribly inefficient for how Lemmy works: each server maintains a copy of each community that it’s users are subscribed to, and changes to those communities need to be communicated across each of those instances.
Given this architecture, it is much more efficient and robust to have users concentrate on what are effectively high performance cacheing servers, and communities spread out on smaller, interest focused instances.
Yeah, I think this is the way things should move in the future. Have community vs user focuses on servers instead of having the same server get hit with both high community/comment usage and a server with lots of login/audit/user browsing requests. Servers with big communities could focus on stability and perfomance. Servers with users could focus on cool UIs and features for their users.
Bro I was talking about you. read my previous comment
