|
1 | | -# Mindscape |
| 1 | +# Mindscape AI — AI Spatial Reasoning & STEM 3D Tutor |
2 | 2 |
|
3 | | -Mindscape turns a static maths or physics worksheet into an interactive 3D lesson. Learners upload a diagram or type a question, inspect and manipulate the generated scene, then explain their reasoning to receive focused feedback rather than just an answer. |
| 3 | +Mindscape turns static STEM worksheets, 3D CAD models, and physics problems into an interactive, real-time 3D playground. Learners upload diagrams or CAD files, explore electric field vector mechanics, manipulate generated scenes, and participate in multi-user real-time 3D classrooms with live AI spatial tutoring. |
4 | 4 |
|
5 | | -The flagship flow is a surface-area lesson: worksheet to labelled 3D solid, then exploration, prediction, explanation, and a follow-up challenge. |
| 5 | +--- |
6 | 6 |
|
7 | | -## Try the judge-safe demo |
| 7 | +## Key Capabilities |
8 | 8 |
|
9 | | -Start the app and open [http://localhost:3000/?demo=true](http://localhost:3000/?demo=true). The guided cuboid surface-area lesson runs through the local planner and does not require API keys. It is designed for a reliable demo recording or judge review. |
| 9 | +### ⚡ 1. Advanced Interactive Physics & Vector Simulations |
| 10 | +- **Coulomb Force Vector Arrows**: Real-time 3D visualization of pairwise electrostatic forces ($F_{ij} = k \frac{q_i q_j}{r_{ij}^2} \hat{r}_{ij}$) with attraction/repulsion color dynamics and magnitude vectors. |
| 11 | +- **Dynamic Field Line Particle Flow**: Interactive RK4 streamline particle flow visualizing electric field lines. |
| 12 | +- **Physics Playground Simulation**: Play/Pause trajectory integration accelerating charges under electrostatic forces. |
10 | 13 |
|
11 | | -## Local setup |
| 14 | +### 📦 2. Custom 3D Model & CAD Importer (.gltf / .glb / .obj / .stl) |
| 15 | +- **Drag & Drop Viewport Uploading**: Learners and engineers can drag custom 3D files directly into the Three.js viewport. |
| 16 | +- **AI Spatial Analysis Metrics**: Automated geometry extraction (bounding box dimensions $dx, dy, dz$, surface area, estimated volume, vertex & face counts) fed directly to the AI Tutor for spatial reasoning. |
12 | 17 |
|
13 | | -### Requirements |
| 18 | +### 👥 3. Multi-User Real-Time 3D Classroom (WebSockets) |
| 19 | +- **Shared 3D Canvas Rooms**: Join via 6-character room code to connect teachers and learners live. |
| 20 | +- **Real-Time Spatial Sync**: Object creation, scaling, rotation, deletion, presence cursor indicators, and tutor hints sync instantly across devices via WebSockets (`wss://`). |
| 21 | + |
| 22 | +--- |
14 | 23 |
|
15 | | -- Node.js 20+ |
16 | | -- npm |
17 | | -- Webcam for hand tracking |
18 | | -- Microphone for push-to-talk voice mode |
| 24 | +## Quick Start & Local Setup |
| 25 | + |
| 26 | +### Requirements |
| 27 | +- **Node.js 22+** |
| 28 | +- **npm** |
| 29 | +- Webcam (optional, for hand tracking) |
| 30 | +- Microphone (optional, for push-to-talk voice mode) |
19 | 31 |
|
20 | | -### Install |
| 32 | +### Installation |
21 | 33 |
|
22 | 34 | ```bash |
| 35 | +# Install dependencies |
23 | 36 | npm install |
24 | 37 | pip install -r requirements.txt |
25 | 38 | ``` |
26 | 39 |
|
27 | | -### Environment |
| 40 | +### Environment Configuration |
28 | 41 |
|
29 | | -Create `.env.local` in the project root: |
| 42 | +Create a `.env.local` file in the project root: |
30 | 43 |
|
31 | 44 | ```env |
32 | 45 | GEMINI_API_KEY=your_google_ai_studio_key |
33 | 46 | GROQ_API_KEY=your_groq_api_key |
34 | 47 | ``` |
35 | 48 |
|
36 | | -The app requires these keys for vision (Gemini), conversational tutoring (Groq), and transcription (Groq). |
37 | | - |
38 | | -## Run |
| 49 | +### Development Server |
39 | 50 |
|
40 | 51 | ```bash |
41 | 52 | npm run dev |
42 | 53 | ``` |
43 | 54 |
|
44 | | -Then open `http://localhost:3000`. Use `?demo=true` for the no-key guided worksheet demo. |
| 55 | +Open [http://localhost:3000](http://localhost:3000) in your browser. Use `?demo=true` for the offline guided worksheet demo. |
45 | 56 |
|
46 | | -## Verify |
| 57 | +### Quality & Test Suite |
47 | 58 |
|
48 | 59 | ```bash |
49 | 60 | npm run quality |
50 | 61 | ``` |
51 | 62 |
|
52 | | -This runs linting and the complete automated test suite. |
53 | | - |
54 | | -## Built with Codex and GPT-5.6 |
55 | | - |
56 | | -Codex and GPT-5.6 were used to stabilize the interactive lesson flow, repair the streaming voice lifecycle, add a no-key demo path, strengthen test coverage, and refine the submission-ready product experience. The project keeps deterministic lesson and scene fallbacks so the core learning journey remains demonstrable when external model services are unavailable. |
57 | | - |
58 | | -## Architecture |
59 | | - |
60 | | - ### High-level system design |
61 | | - |
62 | | - ```text |
63 | | - +---------------------------------------------------------------+ |
64 | | - | Frontend | |
65 | | - | Vanilla JS + Three.js | |
66 | | - | | |
67 | | - | - Question input (text, image, screenshot) | |
68 | | - | - 3D scene rendering | |
69 | | - | - Tutor panel | |
70 | | - | - Voice UI (Browser Web Speech API) | |
71 | | - | - Hand tracking | |
72 | | - | - KaTeX rendering | |
73 | | - | - Real-time "Limited Mode" alerts | |
74 | | - +---------------------------------------------------------------+ |
| 63 | +Runs ESLint and the complete automated unit test suite (170+ tests). |
| 64 | + |
| 65 | +--- |
| 66 | + |
| 67 | +## ☁️ Free Cloud Deployment (Render) |
| 68 | + |
| 69 | +Mindscape AI includes a production multi-stage `Dockerfile` (Node 22) and `render.yaml` blueprint for 1-click free deployment on **Render** with full WebSocket (`wss://`) support. |
| 70 | + |
| 71 | +### 1-Click Render Deployment |
| 72 | +1. Push code to your GitHub repository. |
| 73 | +2. Log into [Render Dashboard](https://dashboard.render.com/) $\rightarrow$ Click **New +** $\rightarrow$ **Blueprint**. |
| 74 | +3. Select your repository. Render will automatically detect `render.yaml`, build Vite production assets, and launch the Node.js + WebSocket Web Service for free. |
| 75 | + |
| 76 | +--- |
| 77 | + |
| 78 | +## System Architecture |
| 79 | + |
| 80 | +```text |
| 81 | ++---------------------------------------------------------------+ |
| 82 | +| Frontend | |
| 83 | +| Vanilla JS + Three.js + WebSockets | |
| 84 | +| | |
| 85 | +| - Question & CAD Input (text, image, .gltf, .obj, .stl) | |
| 86 | +| - Physics Simulation & Coulomb Vector Arrow Overlay | |
| 87 | +| - Multi-User Real-Time 3D Classroom UI | |
| 88 | +| - AI Tutor & KaTeX math rendering | |
| 89 | +| - Hand tracking & 3D gesture controls | |
| 90 | ++---------------------------------------------------------------+ |
75 | 91 | | |
76 | | - | HTTP / SSE |
| 92 | + | HTTP / SSE / WebSockets (ws://) |
77 | 93 | v |
78 | | - +---------------------------------------------------------------+ |
79 | | - | Backend | |
80 | | - | Node.js + Hono API | |
81 | | - | | |
82 | | - | - Request validation | |
83 | | - | - Gemini & Groq Hybrid integration | |
84 | | - | - 3-Tier Model Failover (Gemini -> Groq 70B -> Groq 8B) | |
85 | | - | - Text-as-Vision Fallback (Rescues scene interpretation) | |
86 | | - | - SceneSpec generation (Strict JSON enforcement) | |
87 | | - | - Tutor streaming | |
88 | | - | - Voice pipeline coordination (Whisper STT) | |
89 | | - +---------------------------------------------------------------+ |
| 94 | ++---------------------------------------------------------------+ |
| 95 | +| Backend Server | |
| 96 | +| Node.js 22 + Hono API + WebSocket Room Manager | |
| 97 | +| | |
| 98 | +| - Real-time Classroom Room Broadcasting | |
| 99 | +| - Gemini & Groq Hybrid AI Orchestration | |
| 100 | +| - 3D CAD Geometric Metrics Extraction | |
| 101 | +| - 3-Tier Model Failover (Gemini -> Groq 70B -> Groq 8B) | |
| 102 | +| - SceneSpec generation & Tutor Streaming | |
| 103 | ++---------------------------------------------------------------+ |
90 | 104 | | |
91 | 105 | +--------------+--------------+ |
92 | 106 | v v |
93 | | - +-----------------------+ +-----------------------+ |
94 | | - | Google Gemini | | Groq (Llama 3) | |
95 | | - | | | | |
96 | | - | - Primary Vision | | - Primary Reasoning | |
97 | | - | - Primary Planning | | - Whisper STT (v3) | |
98 | | - | - Failover Chat | | - 70B Scene Planning | |
99 | | - | | | - Visual Failover | |
100 | | - +-----------------------+ +-----------------------+ |
101 | | - ``` |
102 | | - |
103 | | -## Hardened Failover System |
104 | | - |
105 | | -The system is built for hackathon-grade resiliency: |
106 | | -- **Vision Fallback**: If Gemini 2.0 Flash hits quota or fails, the system automatically falls back to **Llama 3.2 11B Vision** on Groq. |
107 | | -- **Interpretation Fallback**: If all vision models fail to interpret a diagram, the system uses the high-capacity **Llama 3.3 70B** to interpret the question text alone, ensuring a valid 3D scene is still built. |
108 | | -- **Search Fallback**: Semantic search (Gemini Embeddings) automatically falls back to **Lexical Keyword Search** if the API key is restricted or out of quota. |
109 | | -- **Transparency**: The UI displays a "Limited Mode" warning in the evidence panel whenever the system is operating in a fallback state. |
| 107 | ++-----------------------+ +-----------------------+ |
| 108 | +| Google Gemini | | Groq (Llama 3) | |
| 109 | +| | | | |
| 110 | +| - Primary Vision | | - Primary Reasoning | |
| 111 | +| - Primary Planning | | - Whisper STT | |
| 112 | +| - Failover Chat | | - 70B Scene Planning | |
| 113 | ++-----------------------+ +-----------------------+ |
| 114 | +``` |
| 115 | + |
| 116 | +--- |
| 117 | + |
| 118 | +## License & Credits |
| 119 | + |
| 120 | +Built with Three.js, Hono, Node.js 22, Google Gemini API, and Groq SDK. |
0 commit comments