Reasoning is already taking place in latent space, the CoT is just a sideeffect

While I was reading the top level message I was thinking on something wrong those lines. That being said I think it's somewhat useful to be able to see the reasoning in a meaningful format. There's already enough hidden layers of unfathomable meaning.

What if instead of a latent they used at a shorter form of note-taking-like reasoning, using more symbol to achieve a denser CoT. We'd get the best of both worlds. WDYT ?

What does ‘latent’ mean in this context?

"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated.

I think https://arxiv.org/abs/2412.06769 is a good description of the premise of reasoning in latent space, although https://arxiv.org/abs/2604.15726 argues it's already what really happens.

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.

It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc.

Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.

> The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.

Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.

More like erasing all other branches just prior to that word from your head, the past thoughts would still remain.

The kv cache is not discretised, only the output tokens.

The computation still has to be done, even if it's more efficient in a higher-entropy space.