Author: Solo Researcher & AI Systems Architect
Technical Whitepaper & PhD-Level Project Report
In STEM education, a persistent cognitive friction exists when representing three-dimensional spatial mathematical and physical systems (e.g., vector calculus, electromagnetism, stereometry) on traditional two-dimensional mediums. This paper presents Mindscape, an intelligent, hands-free 3D spatial reasoning tutor that dynamically generates interactive mathematical environments from natural language prompts or visual worksheets. Mindscape implements a decoupled client-server architecture combining a local computer vision pipeline (running Google MediaPipe WebAssembly) for bare-hand spatial gesture tracking, a client-side WebGL engine (Three.js) for mathematical scene projection, and a hybrid AI reasoning engine on the backend. The backend utilizes a deterministic regex-based mathematical parser alongside high-speed LLM inference (Llama 3 70B via Groq LPUs and Gemini 2.0 Flash via Hono) with strict structural guardrails (Pre-Schema Anchoring and Strict Math Mode) to mitigate hallucinatory proofs. All 169 unit test assertions verified a 100% pass rate with sub-second scene construction latencies.
Traditional STEM pedagogy relies on flat, 2D projections (e.g., whiteboards, textbooks) to convey 3D mathematical realities. This mismatch triggers high cognitive load as students must mentally execute rotational transforms to resolve spatial concepts.
According to Paivio’s Dual-Coding Theory and Sweller’s Cognitive Load Theory, when visual representations align with physical manipulation, semantic integration is significantly accelerated. Mindscape addresses this by converting static inputs into interactive, physical sandbox models.
This project explores:
- AI-Driven Scene Scaffolding: How LLMs can be constrained to output mathematically valid, parseable rendering blueprints.
- Zero-Latency HCI: Mapping bare-hand monocular camera frames to high-fidelity 3D WebGL transformations.
- Hybrid Logic Pipelines: Decoupling deterministic mathematical solvers from generative reasoning modules to eliminate AI latency and hallucination.
graph TD
A[User Input: Text/Worksheet Image] --> B{Complexity Evaluator}
B -- Simple Textbook Query --> C[Deterministic Geometry Parser]
B -- Complex/Multi-part Query --> D[Hybrid AI Planning Engine]
C --> E[3D Scene Specification JSON]
D --> E
E --> F[WebGL Renderer Three.js]
G[Webcam Feed] --> H[MediaPipe WASM Tracker]
H --> I[Raycasting & Interaction Pipeline]
I --> F
F --> J[Interactive 3D Visual Scene]
To minimize API dependency and achieve zero-hallucination for standard vectors, Mindscape utilizes a deterministic regex-based interceptor (analytic.js). When a query matches parametric formats, Javascript computes the geometric components directly.
Given two lines
The algorithm checks for parallel alignment by computing the cross-product:
If
If the system is inconsistent, the lines are skew. The shortest distance
Mindscape computes this instantly, rendering helper points at the closest approach coordinates
The computer vision pipeline runs client-side using WebAssembly (WASM). Google MediaPipe Hand Landmarker extracts
(8) Index Tip (12) Middle Tip
\ /
(7) \ / (11)
\ /
(6) [Knuckles] (10)
\ /
(5) (9)
(4) \ /
\ \ /
Thumb\________\/ (0) Wrist
The Euclidean distance
An active selection (grab/pinch) is registered when:
Coordinates from the monocular 2D camera viewport
To prevent LLM mathematical hallucinations, Mindscape implements two primary prompt guardrails:
To avoid the attention attenuation observed in long-context generation, all logical execution constraints are positioned at the absolute beginning of the system prompt (prompts.js).
Specifically, the model is instructed to solve the mathematical constraints (e.g., verifying vector intersections or performing cross-products) before constructing the scene specification object. This enforces logical consistency before the model allocates tokens for structural JSON keys.
When solving pure algebraic vector calculus, the model enters a restricted mathematical persona. It disables conversational fillers and focuses solely on numeric outputs.
This prevents Generative Collapse, a phenomenon where the LLM repeats "volume is 0" or fails to complete JSON arrays because it attempts to visually interpret 1D/2D lines as 3D volumes.
To bypass API rate-limiting and minimize latency, Mindscape implements a decoupled multi-model architecture:
- Inference & Structure Solver: Powered by Groq's LPUs running Llama 3 70B to generate structured JSON blueprints in less than a second.
- Conversational Assistant: Powered by Gemini 2.0 Flash via Hono stream wrappers (
chatService.js) to provide real-time explanations without blocking the primary structure generator.
+-------------------------------------------------------------+
| Client Request |
+-------------------------------------------------------------+
|
+------------------+------------------+
| |
v v
+-----------------------+ +-----------------------+
| Groq LPU (Llama 3) | | Gemini 2.0 Flash |
| 3D Scene Generator | | Tutor Conversation |
| (JSON blue print) | | (Chat Stream SSE) |
+-----------------------+ +-----------------------+
Mindscape incorporates several defensive programming strategies to ensure stability:
The computer vision loop in app.js runs at 60fps. To prevent application crashes when a hand leaves the webcam view, the tracking logic is wrapped in a dynamic try/catch frame skip block:
try {
const results = handLandmarker.detectForVideo(webcamEl, timestamp);
if (!results.landmarks || results.landmarks.length === 0) {
flushInteractionOverlays();
} else {
processHandLandmarks(results.landmarks);
}
} catch (error) {
console.warn("MediaPipe frame skipped:", error.message);
flushInteractionOverlays();
}When generating plans, Mindscape implements a multi-tier fallback sequence:
[Gemini Vision preferred] ──(Quota Hit)──> [Llama 3.2 Vision on Groq] ──(Failed)──> [Llama 3.3 70B Text-Only Inference] ──(Failed)──> [Local Deterministic Heuristic Plan]
This sequence guarantees that an interactive 3D layout is rendered even during total API outages.
The implementation was validated using automated test scripts and linting tools:
- Linter Validation: The codebase passes
eslint .with zero errors or warnings. - Test Suite Verification: Running
npm testexecutes 169 unit tests with a 100% pass rate in1593ms. - API Capabilities Check: Returns preferred model routes and backup capabilities correctly.
- Voice Pipeline Verification: Tests verify that conversational history is normalized and the Web Speech API strips formatting before speech synthesis.
From a constructivist learning perspective, Mindscape enables Active Inquiry. Instead of receiving static answers, students interact with the concepts:
- Observe: Review the initial geometric coordinates.
- Predict: Formulate a hypothesis (e.g., the intersection point coordinates).
- Check: Move vectors and points using hand gestures to test the hypothesis.
- Reflect: Receive immediate feedback from the tutor chatbot based on the new scene state.
This loop supports deeper visual intuition, helping students master abstract geometries without expensive VR equipment.
Mindscape demonstrates that hands-free, interactive 3D spatial reasoning is achievable using standard web technologies.
Future work will focus on integrating real-time stereoscopic WebXR rendering for head-mounted displays and exploring multi-user collaborative sandboxes for remote classrooms.