Last night, Lemmygrad went down due to a hardware glitch on our VPS provider. Unfortunately, they were unable to do anything to help me fix the issue, and asked that I reinstall from scratch 🙁 . They did inform me that the hardware issue at fault is fixed at least.
I do daily backups, so we lost a day of posts.
Pictures are in the process of being restored now, but it should take a day or so to restore them all.
Let me know if there are any other ongoing issues so I can fix them. Again, sorry everyone.
Whenever Lemmygrad is down I just sigh and say ‘the bourgeoisie at it again 🙄’
Being involved in devops, I can relate to how much this sucks and I appreciate you taking the time to fix it. o7
Thanks comrade, ya these can be nightmare scenarios. I at least have several backup locations, so at most we lose a day’s worth of data.
I’m not involved in this type of work but I would just pull the plug, I can’t understand the amount of patience you need for this shit
Everything will be going fine for weeks, and then: oopsie poopsie everything’s gone!
Anecdotally, my instances have poofed into thin air even on AWS, so this seems not too rare. You use a managed database or something you’ve setup yourself?
Set up ourselves. Its postgres running in a docker container, with daily backups.
The only explanation is that someone posted cringe so big they went to mess with the VPS. Let’s act like it never happened but this better be the last time /s
Honestly how dare you 😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾🤖😾😾😾😾😾😾😾😾😾😾
Our VPS on what went wrong:
I mean, you can’t stop a kitty cat from having fun, don’t be a monster
My cat
Can relate.
we lost a day of posts
Spongebob_suffering.jpeg
Pain
Have this consolation prize:
more pain
No worries, these things happen. Look on the bright side, thankfully it was just one day loss and not more.
At least its back up now, glad you guys were able to fix it.
yay!!! uwu
I’m glad it’s back, thank you for your work!
K, pics are done restoring.
nice to get an update, i was confused yesterday. every time i consistently can’t connect to lemmygrad i assume the worst hahaha
thanks for your hardwork. I was concerned there was some type of ddos or something
On the topic of connection issues. I haven’t been able to connect to the front page when I’m logged in to my account. But i can still visit the communities page or go to specific communities. If i log out the front page works.
Not sure about that one, you might have to open up an issue on the lemmy github.
Will do, thanks.
I was worried that the site got nuked for good because the outage lasted this long, very glad to know it’s not the case! I can imagine the horror you felt when this happened, data loss is a nightmare (I have to deal with this stuff at work). rip to the posts that got lost in the ether but you did what you could.
I couldn’t log in for like 5 days over the summer. I always just assume the issue will get fixed eventually.
Is it feasible to restore the posts and/or comments that were federated to lemmy.ml? I can see my stuff from yesterday there
(oh and also thank you for dealing with this so quickly)
I don’t think they can get sync’d correctly back unfortunately, because the original source is gone.
That’s unfortunate
might have to manually repost 👀
Edit: nevermind, that’s not the reason, but yeah, definitely something weird happening over there, some comments are instantly getting a bunch of votes
Hrm… I’ll take a look.
Is there even 20 people on lemmy?