The Geometry Inside a Language Model
It's not counting. It's moving through geometry.
When we say a model “learns,” we usually imagine weights adjusting.
But this paper suggests something deeper.
The researchers found that character count wasn’t stored as a simple scalar value. Instead, it appeared on what they describe as a low-dimensional curved manifold within the model’s activation space.
Let’s translate that.
Inside a language model, each token generates a very high-dimensional activation vector. Think of it as a point in a vast mathematical space.
What the researchers found is that as the model processes text and “counts” characters, those activation vectors move along a structured path, a curved surface embedded in that space.
In other words:
The model isn’t storing a number in a neat box.
It’s moving through a geometric shape.
Position along that shape corresponds to character count.
Even more striking, specific attention heads appear to perform transformations across this geometry to determine when to insert a line break.
That’s not just surface mimicry.
That’s internal structure.
This doesn’t mean the model has symbolic reasoning in the classical AI sense. But it does suggest that structured computation can emerge through geometry alone.
In the next piece, I’ll explore what this means for interpretability, and why this shifts how we should think about understanding AI systems.
