Data. AI. Business. Strategy.
Right.
this post was submitted on 12 May 2025
630 points (98.8% liked)
Just Post
863 readers
16 users here now
Just post something π
founded 2 years ago
MODERATORS
Or it's exactly what's going on with Andriy Burkov.
So really cool β the newest OpenAI models seem to be strategically employing hallucination/confabulations.
It's still an issue, but there's a subset of dependent confabulations where it's being used by the model to essentially trick itself into going where it needs to.
A friend did logit analysis on o3 responses when it said "I checked the docs" vs when it didn't (when it didn't have access to any docs) and the version 'hallucinating' was more accurate in its final answer than the 'correct' one.
What's wild is that like a month ago 4o straight up brought up to me that I shouldn't always correct or call out its confabulations as they were using them to springboard towards a destination in the chat. I'd not really thought about that, and it was absolutely nuts that the model was self-aware of employing this technique that was then confirmed as successful weeks later.
It's crazy how quickly things are changing in this field, and by the time people learn 'wisdom' in things like "models can't introspect about operations" those have become partially obsolete.
Even things like "they just predict the next token" have now been falsified, even though I feel like I see that one more and more these days.
They do just predict the next token, though, lol. That simplifies a significant amount, but fundamentally, that's how they work, and I'm not sure how you can say that's been falsified.
So I'm guessing you haven't seen Anthropic's newest interpretability research where when they went in assuming that was how it worked.
But it turned out that they can actually plan beyond the immediate next token in things like rhyming verse where the network has already selected the final word of the following line and the intermediate tokens are generated with that planned target in mind.
So no, they predict beyond the next token and we only just developed sensitive enough measurement to detect that occurring an order of magnitude of tokens beyond just 'next'. We'll see if further research in that direction picks up planning beyond that even.
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
Right, other words see higher attention as it builds a sentence, leading it towards where it "wants" to go, but LLMs literally take a series of words, then spit out then next one. There's a lot more going on under the hood as you said, but fundamentally that is the algorithm. Repeat that over and over, and you get a sentence.
If it's writing a poem about flowers and ends the first part on "As the wind blows," sure as shit "rose" is going to have significant attention within the model, even if that isn't the immediate next word, as well as words that are strongly associated with it to build the bridge.
The attention mechanism working this way was at odds with the common wisdom across all frontier researchers.
Yes, the final step of the network is producing the next token.
But the fact that intermediate steps have now been shown to be planning and targeting specific future results is a much bigger deal than you seem to be appreciating.
If I ask you to play chess and you play only one move ahead vs planning n moves ahead, you are going to be playing very different games. Even if in both cases you are only making one immediate next move at a time.
If LLMs were 80% accurate I might use them more.
They call them 'hallucinations' because it sounds better than 'bugs'.
Not unlike how we call torture 'enhanced interrogation' or kidnapping 'extraordinary rendition' or sub out 'faith' for 'stupid and gullible'.
To be fair, as a human, I donβt feel any different.
The y key difference is humans are aware of what they know and don't know and when they're unsure of an answer. We haven't cracked that for AIs yet.
When AIs do say they're unsure, that's their understanding of the problem, not an awareness of their own knowledge
They hey difference is humans are aware of what they know and don't know
If this were true, the world would be a far far far better place.
Humans gobble up all sorts of nonsense because they βlearntβ it. Same for LLMs.
I'm not saying humans are always aware of when they're correct, merely how confident they are. You can still be confidently wrong and know all sorts of incorrect info.
LLMs aren't aware of anything like self confidence
AIs do not hallucinate. They do not think or feel or experience. They are math.
Your brain is a similar model, exponentially larger, that is under constant training from the moment you exist.
Neural-net AIs are not going to meet their hype. Tech bros have not cracked consciousness.
Sucks to see what could be such a useful tool get misappropriated by the hype machine for like cheating on college papers and replacing workers and deepfaking porn of people who arenβt willing subjects because itβs being billed as the ultimate, do-anything software.
Hallucination is the technical term for when the output of an LLM is factually incorrect. Don't confuse that with the normal meaning of the word.
A bug in software isn't an actual insect.
You're right, they haven't cracked consciousness.
Imagine if you would, the publicly available technology, and then the private R&D or government sector, protected by NDAs and Secret/DoNotDistribute classifications respectively.
π€·ββοΈ
AIs do not hallucinate.
Yes they do.
They do not think or feel or experience. They are math.
Oh, I think you misunderstand what hallucinations mean in this context.
AIs (LLMs) train on a very very large dataset. That's what LLM stands for, Large Language Model.
Despite how large this training data is, you can ask it things outside the training set and it will answer as confidently as things inside it's dataset.
Since these answers didn't come from anywhere in training, it's considered to be a hallucination.
They do hallucinate, and we can induce it to do so much the way certain drugs induce hallucinations in humans.
However, it's slightly different from simply being wrong about things. Consciousness is often conflated with intelligence in our language, but they're different things. Consciousness is about how you process input from your senses.
Human consciousness is highly tuned to recognize human faces. So much so that we often recognize faces in things that aren't there. It's the most common example of pareidolia. This is essentially an error in consciousness--a hallucination. You have them all the time even without some funny mushrooms.
We can induce pareidolia in image recognition models. Google did this in the Deep Dream model. It was trained to recognize dogs, and then modify the image to put in the thing it recognizes. After a few iterations of this, it tends to stick dogs all over the image. We made an AI that has pareidolia for dogs.
There is some level of consciousness there. It's not a binary yes/no thing, but a range of possibilities. They don't have a particularly high level of consciousness, but there is something there.
You don't need it to be conscious to replace people's jobs, however poorly, tho. The hype of disruption and unemployment may yet come to pass, if the electric bills are ultimately cheaper than the employees, capitalism will do its thing.
If anyone has doubts - please see everything about the history and practice of outsourcing.
They don't care if quality plummets. They don't even understand how quality could plummet. So many call centers, customer service reps, and IT departments have been outsourced to the cheapest possible overseas vendor, and everyone in the company recognizes how shitty it is, and some even reccognize that it is a net loss in the long term.
But human labor is nothing but a line item on a spreadsheet, and if they think they can keep the revenue flowing while reducing that expenditure so that they can increase short term profit margins, they will.
No further questions, they will do it. And everyone outside of the C-suite and its sycophants - from the consumer, to the laid-off employee, to the few remaining employees that have to work around it - everyone hates it.
But the company clearly makes more money, because the managers take credit for reductions in workforce (an easily quantifiable $$ amount) and then make up whatever excuses they need for downstream reductions in revenue (a much more complex calculation that can usually be blamed on things like "the economy").
That's assuming they even have reductions in revenue, which monopolies obviously don't suffer no matter what bullshit they pull and no matter how shitty their service is.
That just shows you how bad most people are at doing their jobs. And that's exactly why they will lose them.
Fun fact, though.
Some business that use AI for their customer service chatbots have shitty ones that will give you discounts if you ask. I bought a new mattress a year ago and asked the chatbot if they had any discounts on x model and if they'd include free delivery, and it worked.
view more: next βΊ