Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn’t “go rogue.” They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a “human in the loop” that checked each iteration through the Python loop to ensure it hadn’t gone off the rails), they trashed a competitor’s servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous. If you showed up at Defcon and gave a talk about how your autonomous malware did something unexpected and damaged someone else’s computers, the first question from the audience would be “Why are you so shit at making secure sandboxes?” It wouldn’t be “How are you so awesome at making hacking tools?”
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it’s not new bad news. Irresponsible parties have been doing this for years, most notably the NSA…
…
Riley had a very good way of summarizing this: “LLMs are real, AI is fake.” LLMs – chatbots trained on things like CTF logs that can break into servers – are real. They’re on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it.
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.” The Hugging Face hack isn’t a mysterious, supernatural occurrence. It’s a Python loop and a chatbot. The people responsible didn’t accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
It’s fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that – and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That’s a productive kind of worrying, with a chance of addressing your area of concern. It’s infinitely more reasonable than locking yourself in the toilet with a flashlight and saying “Ayyyyy Eyyyyyye” into the mirror until you wet yourself.
You must log in or # to comment.
Oh no, I don’t like the tokens generated by the hand-cranked token generator I won’t stop cranking
Right up there with “why wont Twitter do something about these obscure Nazi accounts I retweet to my audience of 2 million”
Yeah this is the part about all this AGI talk and AI going rogue, it requires a prompt. It responds to an input. These agentic systems have powerful tools that can do other things then spit out text but it’s always responding to a prompt. It may be in a loop responding to its own prompt but it’s still a prompt. And what it spits out is never really anything new. It’s some derivation of what it’s ingested. Actual human intelligence works differently. We do come up with new ideas, new thoughts. So I’ve often thought these LLMs are a dead end to AGI.
it requires a prompt
As delivered to public chat interfaces, yes. They all eventually stop and ask “how am I doing?”
Structurally, theoretically, nothing prevents them from running an infinite loop and continuing to act without that checkpoint. They could be given a goal to “optimize” whatever and just act and act and act in what their pattern-matching systems judge to be actions pursuing “optimal” results.
There was an article about “the genie effect” where protagonists are “tricked” by being given something they technically asked for but not what they really wanted. LLMs carry the additional “risk” of not matching input requests to any recognizable pattern, just following their own path which makes no sense to us. Focused on different priorities.
What we call AI changes every few years… this last round has been particularly brutal because people are subbing in AGI–artificial general intelligence–for AI because LLM’s give the illusion of general intelligence, but it’s a very shallow illusion and it’s deeply embarrassing for humanity how easily we’ve been suckered by it.
LLM AI in the hugging face incident is just a complex roomba, brute forcing every direction until it can get out from under the couch. In fact, if they wanted to demonstrate how SMART their hacking AI is, they would have disclosed how many tokens its burned in the attempt. That they didn’t tell us how many tokens it burned, how many prompts it went through, tells you it probably is not that impressive. It probably tried thousands of different prompts, which is to say it is only a monkey at a typewriter.
LLM AI in the hugging face incident is just a complex roomba, brute forcing every direction until it can get out from under the couch.
That is such a great statement, I’m going to have to use that when describing LLMs.
Thanks 👍
Honestly, I can’t get anybody to give me a good definition of intelligence, artificial or otherwise. Once we can define that in concrete terms, maybe we can finally compare the two.
Not a definition of intelligence, but something I often come back to is what it does at rest.
To the best of my understanding of how LLMs work, if you’re not feeding the thing inputs, the program is doing absolutely nothing. It isn’t curious and doesn’t seek new information or stimuli. It effectively is just on pause until the next prompt. There is nothing akin to an inner monologue or thought process happening in absence of what is fed to it.
I’m not up on all of the epistemology or theory of mind type stuff, but I’d imagine some might argue about that distinction in humans; we are always processing input in some form as the input from our senses is never shut off until we’re dead. But I think the “at rest” case is a relevant distinction that does get at motivation and real thought born of real intelligence.
The way the systems operate right now, your session is time-sharing the hardware with all the other sessions.
Conceivably, you could set background tasks to “contemplate, research and develop” concepts related to the central task, but those tend to burn a lot of tokens relative to the useful output they provide.
If you think of an LLM agent like a butler, you really don’t want your butler getting ideas of their own and acting on them while you’re ignoring them, you just want them available when you call on them. To broaden the analogy, embodied in physical robots, LLM agents could be set upon endless tasks like maintaining a house and grounds, maybe a garden, maybe some farm animals, and each other, with spare capacity leftover to carry out special tasks when called upon.
Agreed, I believe “I think, therefore, I am.” sums that up well.
So much this. They’ve basically figured out that if they burn enough money, they get the equivalent of billions of meth-charged monkeys on typewriters.
Anything with a security hole (so… basically everything) will eventually be broken into if you throw a data center worth of computational power at trying every conceivable hack. That’s basically what they were doing here. The only mildly impressive part about it is the amount of resources they’re willing to throw at this.
I’d say the most impressive part is Sam Altman not being in jail, but that’s more the social engineering of wealth than the technical engineering of scientists.
Rewatch Willy Wonka and the Chocolate Factory, from 1971 (55 years ago) - in it, they have a Siemens System 4004 computer fictionally conversing with a golden ticket seeker… that was the concept of AI 55 years ago: something a “computer expert” could translate for investors.
Today, AI “speaks for itself” and even does a reasonable job of voice recognition and transcription, in hundreds of languages. But still, that’s not good enough. It still can’t tell you where the remaining golden tickets are.
it’s deeply embarrassing for humanity how easily we’ve been suckered by it.
Humanity has been embarassingly quick to blame “the computer” for all manner of problems ever since there have been computers. Many people were suckered by Eliza in the 1960s, current LLMs are many orders of magnitude more complex than Eliza…
It probably tried thousands of different prompts, which is to say it is only a monkey at a typewriter.
Even if it did, it did that self-directed, which is one of the LLM agents’ superpowers: the ability to hammer with brute force attacks quickly and cheaply. There are legitimate applications for this, essentially it’s what Folding At Home and similar projects have been pursuing for many years now.
Remember: every time you repeat a story about how awfully, terribly dangerous their products are, you help them raise more investment capital, which is a key input for their business (hooking up statistical engines to money-furnaces):
Fuck both of them. How about we not build them to prevent more destruction to the ecosystem.
The people in charge have amassed a tremendous investment base, bigger than .com, they can’t continue to live off the skim if they just hand all the money back to the investors. The investors aren’t interested in stemming their losses, they invested because they believe in “the big win,” and a lot of them are so old they really don’t care about what the mess will look like 20 years from now, they just want to go out on top, or die trying.
One of the fun things about this latest cycle of AI is how the goalposts move. Not the ‘AGI in six months’ one. That might as well be mounted on top of a golf cart with a brick laid on the accelerator. But the one that talks about the Total Addressable Market (TAM) and goals of a product.
One of the things they tell people pitching VCs is that you should be addressing a reasonably big TAM (potential customers) and have sensibly large, yet attainable stakes. With AI, though, the TAM was always questionable. Who would want or pay for a chatty ‘bot,’ especially one that ‘hallucinated’ answers? So they pivoted to change the TAM and the stakes. Your personal assistant. No? Therapist? No. Project Manager? Teacher? Still no?
What we have now is a constant churning of THOSE goals. Now, the TAM is pretty much the whole world, and the stakes are either massive unemployment, widescale industrial shutdown, or destruction of the human race. It’s all getting silly. Once you’ve extracted all the investment money you can by maxing out the stakes and scaring the pants off people, there’s not much higher to go.
Meanwhile, all your actual income is coming from pedestrian applications, like summarizing emails, screening job applicants, or coding assistants. There’s just not enough revenue to cover all the costs you’re incurring to get it to hallucinate a little less. So you have to pump up the stakes. Next up: setting off all the nukes! Shutting down the sun and the moon! Inviting alien colonists!
Speaking of going bigger: https://youtu.be/R8SeysuMpA0
Can I just point out how fucking insane it is that “coding assistant” is now considered “pedestrian”?
(Depending of course on what exactly is meant by the term).
Gotta get creative with the power scaling. Next up, solar system! LLMs will destroy the galaxy! No; the whole universe! The multiverse! Or worse?!
This article seems to be yet another one of the thousands of “AI doesn’t really think because we understand how it works” slop pieces.
If you think Cory Doctorow is “slop” you should really consider simply not ever posting anything online again.
Sorry I offended your blogger you have a parasocial relationship with. 🤣
That’s a lot of syllables… Don’t strain yourself.
I read the first third or so of the article and so far the person you’re replying to is correct. Doctorow explains the way LLMs work-- so badly as to be not just misleading but also incorrect in some cases-- as if that somehow invalidates the actions taken by the AI system. They also show signs of pretty extreme dogma to the point of including irrelevant info (eg. Python being easy to use) that serve only to emptionally manipulate the reader. The rest of this article isn’t worth my time, if this changes in later parts of the article let me know and maybe I’ll finish it.
confidently diagnosing an essay as dogma while admitting you didn’t even read the thesis is peak internet.
His mentioning of python being a basic/easy to master language isn’t manipulation, he’s demystifying. It strips the magical hyperbolic framing that has been attributed to AI (and to which those with most to gain from it financially very much wish to feed into!)
His point about being wrapped in python isn’t a failed attempt to give a lecture on transformer architecture, its to explain that the automation loop is controlled by basic inputs from a user controlled and relatively simple software stack, not some sentient machine with a mind of its own. Its not emergent behavior, it’s optimizing execution patterns embedded in the data it was trained on.
For my part, I’m just awed how many comments I come across that are just half-assed lazy people who evidently can barely read, but think they’re experts. Calling Doctorow’s work ‘slop’ or quitting reading while feeling so confident proves his point: people would rather buy into hype than actually understand how the tech works.
:::
It’s incredibly obvious from the first part of the article that Doctorow is using the strongest rhetorical techniques they can, regardless of the technical merit of the resulting arguments. This makes it dogmatic, yes. That’s just kind of how Doctorow is, and always has been. I don’t much care for that method of communication because it is so often used to mislead people with oversimplified emotional language and convince them without being logically sound.
It wouldn’t be as bad if Doctorow’s understanding of the subject matter was at least accurate, but their conclusion is irrelevant if their understanding is unsound. Their main mistake is saying it is essentially doing database lookups while ommitting the entire effect that enables it to appear intelligent, that being compression of the input data into a semantic representation. Anyone familiar with the Compression is Intelligence paper and related work would know that ommitting that step is presenting a narrative so inaccurate that it is not only misleading but actually false. This leads me to conclude that Doctorow either does not understand how AI works, or does understand it but is willing to lie to their audience. Either is unacceptable and renders the conclusion of their argument worthless. Again, if they do more than “demystify” AI by incorrectly explaining how it works then let me know and I will read the rest of the article.
Edit: To be clear, an article actually demystifying AI would be a good thing. My point is that it is a bad idea to overcorrect for the ai bro hype bubble by explaining it just as badly in the other direction.
Pretty sure Cory’s pronouns are he/him.
And he’s been consistently spot on from fighting the good fights against DRM, for right to repair, interoperability, anti-surveillance, against enshitification, and his take here today on AI. I was personally happy to read this because it exactly echoes what I’ve said countless times myself and I’m very glad to be in his good company.
He does not even try to present or debate “how” “ai” works. He’s speaking about how it is employed and what the boundaries are around the rhetoric we hear every day in the press that amounts to nothing more than fear mongering.
There’s absolutely a LOT to be afraid about with respect to how “ai” is used to surveil and control our society, but it has nothing to do with intelligent software and everything to do with corporate and government power, enabled by a mystique created around what is really a (admittedly powerful and useful) statistical parlour trick.
Pretty sure Cory’s pronouns are he/him.
Sorry, force of habit.
I agree with the whole anti-surveillance/anti-corporate-control argument. I think it’s a worthy fight to participate in. I just don’t think it’s wise to go about it in a way that could prove counterproductive by causing people to have inaccurate beliefs.
Edit: After finishing the article, my opinion has softened a bit but is much the same: Doctorow points out an important perspective (this is one of few articles pointing out the security/responsibility side of things, for example), but some of the technical info is unsupported/inaccurate. Overall probably a positive article when taking into account its role in general AI discussion, but it is not entirely accurate on its own.
sTocHastIC ParRoT
