> Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.
Their ability to generate facts is a consequence of the word statistics they are truly using. I think it's fair to say the statistics explanation is a more accurate interpretation.
More accurate than what? Statistics "explanation" is something which is technically correct. But it also doesn't present the full picture — for example, the fact that LLMs clearly build an internal mental model of the question they're talking about.
They build internal representations of the input only, and to the extent needed, to get the statistics right. This really isn't a "world model" of factual data, but rather a "source model" of what would various sources (training texts) say.
The responses of the model don't represent what it understands per some internal model, because there is no "it", only models of the sources it was trained on, and it'll just as happily generate lies as truths, or smart vs dumb answers (it's all just words) if that is what its source modelling calls for.
What most people mean when they say the model is hallucinating/bullshitting isn't where it has learnt a lie, but rather where is is operating "out of distribution", and is therefore (unknowingly) generating a mashup from multiple only loosely related/matching source contexts.
I'm not convinced at all. The only thing they are doing is perturbing some of the model weights in intermediate layers, and seeing if the output of the final layer is consistent with the perturbations. It would be a shitty model if that was not the case.
The fancy part in the paper is figuring out how to perturb the intermediate layers in the way you want. But the findings are not impressive.
Note also that the "probe geometry" stuff is so speculative they left it out of the academic paper completely.
In the same way, it has been known since the 90s that if you take the matrices from Finite Element Models and visualize them as graphs, structures appear that kind of resemble the physical appearance of the object being modelled. Here for instance is for a helicopter:
"Yet nobody thinks Finite Element Models have an internal mental representation of the world."
At this point, I'm not sure some wouldn't argue that.
The difference is, put the AI on a loop, with constant feedback, learning.
Instead of just a 'pre-trained' model. Make the actual model, live, always learning, so the context window is infinite. This of course would not be for everyone, because it would take all the resources of the training infrastructure to be focused on one person/view. But that gets closer to the human mind, and at that point, we probably couldn't say for sure that the 'perturbations' aren't experiencing something subjective.
Where is the proof that humans have an internal mental representation of the world.
No, you're not. Are you genuinely trying to suggest that LLMs, which can:
- Construct arbitrary text that isn't just grammatically but semantically coherent
- Derive intent, subtle intent, from user queries and responses
- Emulate endless different personalities and their reactions to endless stimuli
- Describe in detail the statics and dynamics of the world, including sight, smell, touch and sound
do not have a model of the external world? What do you think a "corpus" means in this context? How is the "corpus" of sensory and evolutionary data that makes you up in any way different?
LLMs are excellent common sense reasoners, and they generalize just fine. Why exactly do you think they get things _subtly_ wrong? Make up API syntax that looks sensible but isn't actually implemented? In order to make these guesses they need to have generalized, they need an understanding of the structure underlying naming, such that they can produce _sensible_ output even if they lack the hard facts.
You are correct.
We are flooded with studies on AI now, so can't find reference.
But just few months ago, saw example of AI, from video, building an internal representation of the world. An internal model of the world. Everyone saying this can't be done, it already is. Maybe can argue it wasn't an LLM, and then I'd say were nitpicking over which technology can do it or not. We already have example of tying them together, symbols and LLM's.
I believe they used the game to show how the same underlying technology would build a mental model not because the game was perfect-information, but because it was easy to probe for without a lot of other unrelated concepts getting in the way.
> In the “mind” of Claude, we found millions of concepts that activate when the model reads relevant text or sees relevant images, which we call “features”.
> One of those was the concept of the Golden Gate Bridge. We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.
Sounds like a mental model to me! An internal representation of an external concept which exists in the real world.
I don't really know if using an "experiment" is the right way to test this?
Like, if you want to test such hypothesis you should first prove that these LLMs were created with that specific function in mind, we are not observing a natural phenomena, it is a manmade(manmade...loosly) algorithm .
You don't know the internal architecture of the system and you are testing in a reality narrow situation, this is basically the scientific method applied incorrectly.
That's a false dichotomy.