β₁ Hole Direction Extraction — Summary
What is a β₁ hole?
In the 4096-dimensional latent space of an LLM (Llama 8B), persistent homology detects β₁ holes — topological loops formed by data points. These loops act as walls that bound the
model's representational distribution.
What does "direction extraction" mean?
The experiment extracts a 4096-dimensional vector that points perpendicular to the wall — the direction you'd need to move to pass through (or over) the hole.
Pipeline:
- Collect ~30-44 embeddings per prompt (prefix accumulations + suffix variations)
- Run persistent homology (Ripser) to detect β₁ holes and their representative cycles
- Compute local PCA on cycle points → find the 2D plane where the hole lives
- Find the direction in the orthogonal complement with minimum variance (most empty)
- Result: a passage direction vector in full 4096-dim space
What does "orthogonality = 0" mean?
The dot product between the passage direction and the cycle plane vectors is exactly 0. This confirms the extracted direction is perfectly perpendicular to the wall — a geometric
validation that the direction truly points "through" the hole, not along it.
What does "dim 940/1917 recurring" mean?
Out of 4096 neurons, dimensions 940 and 1917 have the largest components in the passage direction vector — and they appear repeatedly across different prompt types (reasoning,
creative, etc.). This suggests these two neurons are core boundary-forming neurons in the model, structurally involved in defining the edges of what the model can express.
The 2D wall intuition
In 2D, a β₁ hole is a closed loop of points — a ring. It acts as an impassable wall:
● ─ ● ─ ●
/ \
● (hole) ● ← cycle points forming a wall
\ /
● ─ ● ─ ●
You cannot cross from inside to outside without breaking through the ring. But in 3D, a third dimension opens up — you can jump over the wall by moving perpendicular to the plane:
2D plane (where the cycle lives)
─────────────────────
| ● ─ ● |
| / \ |
| ● hole ● | ← trapped in 2D
| \ / |
| ● ─ ● |
─────────────────────
↑
│ passage direction (perpendicular)
│ move along this axis to bypass the wall
In this experiment, the cycle sits on a 2D plane within 4096-dimensional space. The passage direction is the optimal perpendicular escape route through the remaining 4094
dimensions — and neurons 940 and 1917 contribute the most to that escape.
Analogy: Like being trapped in a 2D maze. The walls are impassable in 2D, but if you can jump into 3D, you simply step over them. This experiment finds exactly which direction to
"jump" in 4096-dimensional space.
β₁ Hole Direction Extraction — Summary
What is a β₁ hole?
In the 4096-dimensional latent space of an LLM (Llama 8B), persistent homology detects β₁ holes — topological loops formed by data points. These loops act as walls that bound the
model's representational distribution.
What does "direction extraction" mean?
The experiment extracts a 4096-dimensional vector that points perpendicular to the wall — the direction you'd need to move to pass through (or over) the hole.
Pipeline:
What does "orthogonality = 0" mean?
The dot product between the passage direction and the cycle plane vectors is exactly 0. This confirms the extracted direction is perfectly perpendicular to the wall — a geometric
validation that the direction truly points "through" the hole, not along it.
What does "dim 940/1917 recurring" mean?
Out of 4096 neurons, dimensions 940 and 1917 have the largest components in the passage direction vector — and they appear repeatedly across different prompt types (reasoning,
creative, etc.). This suggests these two neurons are core boundary-forming neurons in the model, structurally involved in defining the edges of what the model can express.
The 2D wall intuition
In 2D, a β₁ hole is a closed loop of points — a ring. It acts as an impassable wall:
You cannot cross from inside to outside without breaking through the ring. But in 3D, a third dimension opens up — you can jump over the wall by moving perpendicular to the plane:
In this experiment, the cycle sits on a 2D plane within 4096-dimensional space. The passage direction is the optimal perpendicular escape route through the remaining 4094
dimensions — and neurons 940 and 1917 contribute the most to that escape.
Analogy: Like being trapped in a 2D maze. The walls are impassable in 2D, but if you can jump into 3D, you simply step over them. This experiment finds exactly which direction to
"jump" in 4096-dimensional space.