LLMs are trained to do one thing: produce statistically likely sequences of tokens given a certain context. This won’t do much even to poison the well, because we already have models that would be able to clean this up.
Far more damaging is the proliferation and repetition of false facts that appear on the surface to be genuine.
Consider the kinds of mistakes AI makes: it hallucinates probable sounding nonsense. That’s the kind of mistake you can lure an LLM into doing more of.
Anthropic is building some tools to better understand how the LLMs actually work internally, and when they asked it to write a rhyme or something like that, they actually found that the LLM picked the rhyming words at the end first, and then wrote the rest using them at the end. So it might not be as straight forward as we originally thought.
you can poison the well this way too, ultimately, but it’s important to note: generally it is not llm cleaning this up, it’s slaves. generally in terrible conditions.
LLMs are trained to do one thing: produce statistically likely sequences of tokens given a certain context. This won’t do much even to poison the well, because we already have models that would be able to clean this up.
Far more damaging is the proliferation and repetition of false facts that appear on the surface to be genuine.
Consider the kinds of mistakes AI makes: it hallucinates probable sounding nonsense. That’s the kind of mistake you can lure an LLM into doing more of.
Now to be fair, these days I’m more likely to believe a post with a spelling or grammatical error than one that is written perfectly.
I’m not smart enough to spot the error in your comment, so I guess you’re an AI.
Anthropic is building some tools to better understand how the LLMs actually work internally, and when they asked it to write a rhyme or something like that, they actually found that the LLM picked the rhyming words at the end first, and then wrote the rest using them at the end. So it might not be as straight forward as we originally thought.
you can poison the well this way too, ultimately, but it’s important to note: generally it is not llm cleaning this up, it’s slaves. generally in terrible conditions.