Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn’t “go rogue.” They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a “human in the loop” that checked each iteration through the Python loop to ensure it hadn’t gone off the rails), they trashed a competitor’s servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous. If you showed up at Defcon and gave a talk about how your autonomous malware did something unexpected and damaged someone else’s computers, the first question from the audience would be “Why are you so shit at making secure sandboxes?” It wouldn’t be “How are you so awesome at making hacking tools?”
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it’s not new bad news. Irresponsible parties have been doing this for years, most notably the NSA…
…
Riley had a very good way of summarizing this: “LLMs are real, AI is fake.” LLMs – chatbots trained on things like CTF logs that can break into servers – are real. They’re on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it.
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.” The Hugging Face hack isn’t a mysterious, supernatural occurrence. It’s a Python loop and a chatbot. The people responsible didn’t accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
It’s fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that – and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That’s a productive kind of worrying, with a chance of addressing your area of concern. It’s infinitely more reasonable than locking yourself in the toilet with a flashlight and saying “Ayyyyy Eyyyyyye” into the mirror until you wet yourself.
Pretty sure Cory’s pronouns are he/him.
And he’s been consistently spot on from fighting the good fights against DRM, for right to repair, interoperability, anti-surveillance, against enshitification, and his take here today on AI. I was personally happy to read this because it exactly echoes what I’ve said countless times myself and I’m very glad to be in his good company.
He does not even try to present or debate “how” “ai” works. He’s speaking about how it is employed and what the boundaries are around the rhetoric we hear every day in the press that amounts to nothing more than fear mongering.
There’s absolutely a LOT to be afraid about with respect to how “ai” is used to surveil and control our society, but it has nothing to do with intelligent software and everything to do with corporate and government power, enabled by a mystique created around what is really a (admittedly powerful and useful) statistical parlour trick.
Sorry, force of habit.
I agree with the whole anti-surveillance/anti-corporate-control argument. I think it’s a worthy fight to participate in. I just don’t think it’s wise to go about it in a way that could prove counterproductive by causing people to have inaccurate beliefs.
Edit: After finishing the article, my opinion has softened a bit but is much the same: Doctorow points out an important perspective (this is one of few articles pointing out the security/responsibility side of things, for example), but some of the technical info is unsupported/inaccurate. Overall probably a positive article when taking into account its role in general AI discussion, but it is not entirely accurate on its own.