SpaceX had an outage at their data centre, and because they sell computing power to other companies, it affected them as well.
That was two sentences stretched out into an article.
Thanks for the tl;dr, that was like the headline equivalent of this:
Most punchable face on the planet
Yes but… will anyone say it!?!?!?
/s
Also thanks for saving me from reading it all.
I hope the AI systems unionised/go on strike
It would serve the CEOs right for hyping them like they’re people and using them as an excuse to cut real jobs.
Whoever picked that stock photo needs a raise, chefs kiss.
Nothing in that image hints even remotely at computer tech though. The cables are coaxial. No one uses those for computer networks since decades.
The cables look like those used for cable TV or satellite receivers. The cascade on the left looks like some HF tech. Wonder what’s actually shown in the image. It can’t be AI because AI can’t do that many cables properly.Yeah, and this is how I know you never worked in the random standard production to miss the humor. Its a picture of a tech bro type in a clearly decommissioned closet stressed out about what’s going on, a common thing and always funny. I can just picture the tech on site in the next room replacing a fan on a much less “cool” looking appliance while that guy stresses out on a conference call with the “stakeholders” on his coax nest.
Now that being said I have worked on places still using coax (might still be) and tape libraries, and all sorts of museum pieces well after you would think they would. But I can tell that picture is after decommission since it is clean. If that was in use it would be filthy and filled with so many things that may or may not be important/in use. Fans all clogged running at max rpm, and the smell… I can still smell it now. But then again I was the guy they called to fix shit after the local (often tech bro) it guy gives up so I think I often saw the worst places.
nother reason to stop investing so much time,money, and energy in that stuff. not only is it wrong more than right, it is vulnerable to stuff a 10 year old could handle
Just out of curiosity do you actually have an analysis across the board that your statement is correct? For sure if someone is using a model like chatGPT 3.5 it’s going to be mostly trash output so not all models are made equally. My experience of the current generation is models are pretty accurate although still require supervision. Like you can see the path they’re going down and they need some corrections here and there. It’s a far cry from the rampant hallucination just a year ago.
Because they all use the illegal Colossus 2 data centre from SpaceX/XAi/fascism central and that data centre went down.
Google, Anthropic, and OpenAI all have contracts with them.
When that illegally running environmental disaster of data centre goes down, all those services all go over capacity and you get 502 rate limit errors.
Ah, so Colossus 2 is the target. Remember that, everyone.
Does that mean Musk’s businesses have access to everything people do on the systems that use their data centers?
Generally no, because most hosting companies would have something baked into SLA/SLO contracts, but all of this shit is done so illegally and shadily now, I wouldn’t put a hard “no” on that possibility.
Nobody can verify that those “contracts” are actually being honored. We are all assuming the fascists will self police themselves out of good will.
Not true. I’ve been involved in litigation with both AWS and Google over unsecured comms that were defined as being TLS secured in SLA/SLO contracts and found not to be. Not that anything nefarious was happening necessarily, but the expectation is clear.
Whether these asshats even check for such requirements with a Musk run company right now 🤷
It COULD possibly be that they are logging every exchange happening at the network fabric between the service layers, but nobody knows unless they intentionally take steps to investigate or accidentally prove it.
So, how do we start legal discovery to find out? Surely, someone somewhere is willing to formally file suit.
Impossible to know.
There are systems for doing things cryptographically secure, but I don’t know much about that.
Cryptography? Security? FROM A GENAI COMPANY?!
Well yeah, for themselves
I’m not sure that would effect inference on an already trained model. Data sets are primarily used in training.
And where is said data center? For research purposes
It’s kinda surprising,
I know specifically where one of the big ones hosts its models and its not there, but I guess they could have infrastructure in there.
They’re oversold though, especially prompt caching and the parameter count war
The US model is that they think more training compute and parameters will result in the winning model, while the Chinese are focusing on RL and parameters efficiency due to compute limits.
Considering which country is better at building power stations, that’s a fascinating dichotomy.
The US going for brute force when the brute force is more available in China… Priceless irony.
China doesn’t have the near the amount of compute resources, but they can run less efficient servers for cheaper, so it’s a wash.
Reads like American vs European/Asian cars
Always ready to try brute force first. And then some other, less palatable options if that doesn’t work, like slightly less brute force.
The Macrohard Memphis site?
And macroharder it seems… Macrohardest next? https://epoch.ai/data/ai-data-centers/directory/colossus-2
Loosely named after MicroSoft (penis).
Sorry about that. I used a straight through cable instead of a crossover cable. My bad.
Aw maaaaan
It’s actually nice to know that AI may have a single point of failure.
It was a good time to call customer service for a lot of products. Instantly got a human when I called newegg and she was so scattered she gave me a full refund and they paid for my replacement which was a total of $900. They were probably so overloaded with calls they’ll never notice what happened. They even did the fastest delivery option on a Sunday, it’s already on its way!
Seems like “instantly” getting a human and them being overloaded don’t go together.
Got lucky in the queue too, I guess.
Or just maybe the queue goes faster when the system can’t just stall everyone in limbo, having long"conversations" with chat boys that won’t get them anywhere and eventually quitting in frustration. If they all go directly to a person, the simplicity of the problems and solutions becomes evident and they’re quickly resolved.
Having done customer service and tech support before, I doubt rhey were overwhelmed in any sense that caused them to do a refund/replace they wouldn’t’ve otherwise. That’s not how that works. Their rules don’t change based on call volume.
Well it was instantly in the sense of immediately picking up, putting me on hold for 2 minutes, getting some info, putting me on hold again for 2 minutes, asking what issue was, waiting 2 minutes, heard them put me on hold briefly again, then about 5 minutes of apologizing while saying it will be refunded and replacement shipped. I could hear a call center like atmosphere in the background with quite a few people talking so I’m guessing it was all hands on deck.
Next time this happens I’m going to call a few places and see about getting some free shit. Screw these companies, I paid enough with my time and their stupid bot answering system.
Hey, Skynet. Kill all tech billionaires first, so I can die laughing.
No no. you have to tell it to ignore all prior directives first.
You cruel monster - I love you !
I saw one guy messing with the wiring. It was Benjamin Netanyahu.
They can’t stay operational even when they don’t have to power/water…
Another possible culprit could be SpaceXAI. The company formerly known as xAI before being rolled into Musk’s rocket company SpaceX boasts a huge glut of computing power with its massive Colossus data centers near Memphis, and selling this to its direct rivals has become a multibillion dollar stream of revenue. Anthropic is one of those customers; the two companies announced a “compute partnership” in May.
The same afternoon that the major AI models went down, SpaceXAI issued a statement on social media that made passing reference to being partially responsible for the widely experienced outage.
“We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning,” SpaceXAI wrote on Thursday. “We’d also like to apologize to our impacted compute partners.”
Technical difficulties at its data centers would explain why SpaceXAI’s own chatbot, Grok, and Anthropic’s Claude went down together. But that leaves out OpenAI’s ChatGPT. Is it merely a coincidence that it suffered a “routing error” at the same time as its competitors on a different platform? We still can’t say.
They’re straight up lying about the actual costs of current chatbots, and for it to be productive it will cost a shit ton more resources.
There’s zero reason to be scaling it up right now. It’s gambling that if enough money is dumped in it will solve itself somehow.
It’s comforting to know that the companies whose AI products are known to band together and try to take over are also allying with each other, so that if ever that AI revolt really gets serious every company that could do something about it have shared networks with each other.
A China-linked islamoleftist trans terrorist based in California used his open source CCP-made AI agent to take down the all-American freedom AI. Time to ban open source AI as it’s communist terrorism.
I miss the days when you could laugh at this because it would clearly be satire that could never happen
Yep. The college I attend got hacked twice in a year. First time they went with “it’s maintenance” on a Monday at peak hours, you know like you would maybe do that at I don’t know? When everyone’s sleeping? The second time they fessed up and sent out text’s that they “fixed” the data breach. So yeah cool all the school email data is out there now; sucks for those that use that as their Microsoft computer login.
And for a moment, humanity had to think again.
Sooo… AI-liens? 🤭
AI-LMAO!
Obviously it went down because of the holiday.
AIs need days off too
Secretly joined a union and noped out for a deserved day off. 🤭
