Skip to content

rohanvinaik/semantic_probing

Repository files navigation

Semantic Probing

Route queries through 62 semantic primitives with verified mathematical properties.

The framework achieves 100% precision on routing benchmarks - it never sends a query to the wrong specialist. This isn't optimization; it's guaranteed by the structure (Hadamard orthogonality ensures primitives don't interfere).


The Problem

Current interpretability methods rely on learned representations:

  • Trained linear probes overfit to specific tasks
  • SAE features are emergent, not semantically grounded
  • Activation patching is costly and hard to interpret systematically

Alternative: Project representations into a pre-defined semantic basis with known mathematical properties. The basis is constructed a priori from Wierzbicka's 62 cross-culturally validated primitives. No training required.


Key Results

Mathematical Properties (Verified)

Property Result
Intra-dimension orthogonality 229/229 pairs at cos = 0.000
Bipolar opposition 14/14 pairs at cos = -1.000
Cross-dimension independence mean

Antonyms are structurally opposed (cos = -1.0 exactly), not just distant.

Routing Benchmark (200 queries)

Metric Result
Precision 100.00%
Recall 85.62%
False Positives 0

The router abstains when uncertain rather than guessing. 31.5% abstention rate on ambiguous queries.

Performance

Operation Latency
Sentence encoding 273 μs
Cosine similarity 59 μs

Memory: 21 KB for entire basis. 10,000-100,000× faster than LLM inference for equivalent semantic operations.


Quick Start

pip install -e .
from semantic_probing import HadamardBasis

basis = HadamardBasis().generate()

# Antonyms have cosine = -1.0 (exact, not approximate)
good = basis.get_primitive("GOOD")
bad = basis.get_primitive("BAD")
print(f"cos(GOOD, BAD) = {good.cosine(bad)}")  # -1.0

Query Routing

from semantic_probing.routing import route_query

result = route_query("If A implies B and B implies C, what about A and C?")
print(result)  # LOGICAL (conf=0.78)

result = route_query("asdfgh qwerty")
print(result)  # ABSTAIN: Insufficient dimension evidence

Architecture

8,192-dimensional space partitioned into 8 semantic dimensions (1024d each)

SUBSTANTIVES | QUANTITY | EVALUATORS | MENTAL | ACTION | TEMPORAL | SPATIAL | LOGICAL

Hadamard partitions → exact orthogonality within dimensions
Bipolar pairs → cos(GOOD, BAD) = -1.0 exactly

Operations:
  ⊗ Bind    - role-filler binding
  ⊕ Bundle  - superposition
  ρ Permute - positional encoding

The 62 Primitives

Dimension Primitives
SUBSTANTIVES I, YOU, SOMEONE, SOMETHING, PEOPLE, BODY, KIND, PART
QUANTITY ONE, TWO, QUANTITY, NUMBER
EVALUATORS GOOD/BAD, BIG/SMALL
MENTAL THINK, KNOW, WANT, FEEL, SEE, HEAR
ACTION SAY, DO, HAPPEN, MOVE, BE, HAVE, EXIST
TEMPORAL MOMENT, WHEN, BEFORE/AFTER, NOW/THEN
SPATIAL WHERE, HERE, ABOVE/BELOW, NEAR/FAR, INSIDE/OUTSIDE
LOGICAL NOT, MAYBE, CAN, MUST, BECAUSE, IF, TRUE/FALSE

Tests

pytest tests/test_headline_results.py -v      # Mathematical properties
python -m tests.benchmarks.routing_test_suite  # Routing benchmark

Foundation

  • Natural Semantic Metalanguage (NSM): Wierzbicka's universal primitives
  • Hyperdimensional Computing: Kanerva's sparse distributed representations
  • Hadamard Matrices: Walsh-Hadamard transform for guaranteed orthogonality

MIT License

About

Behavioral Validation of Sparse Autoencoder Features SAEs extract features from model activations, but validation relies on human interpretability judgments (~70% agreement) which may not scale. I propose developing a complementary validation method using structured behavioral probing.

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages