Quick answers to the most common setup, runtime, and contribution questions for SpectraX — the AI-powered fitness tracker built for GSSoC'26.
If your question isn't covered here, please open a Discussion or check the Issues tab before filing a new one.
- 🧭 General
- ⚙️ Prerequisites & Environment Setup
- 🚀 Installation & Running Locally
- 🎥 Camera Permissions & Browser Access
- 🧠 WASM, MediaPipe & ML Model Loading
- 🐛 Common Runtime & Development Errors
- 🏗️ How SpectraX Works (For New Contributors)
- 🤝 Contributing & GSSoC'26
- 📚 Related Documentation
SpectraX is an AI-powered fitness companion that uses MediaPipe Pose Detection and Three.js to:
- Track exercises in real time through your webcam
- Count reps automatically (squats, pushups, and more)
- Score your form and give live posture feedback
- Render a 3D skeleton of your movement using WebGL
- Sync session data between frontend and backend over WebSockets
It is a full-stack web app: a React + TypeScript + Vite frontend and a small Express + Socket.io backend.
Any modern desktop browser with support for:
- The MediaStream API (
getUserMedia) - Web Workers
- WebGL (required by Three.js)
Tested combinations: latest Chrome, Edge, Firefox, and Safari on desktop. Older browsers (Internet Explorer, very old Safari versions) are not supported.
Mobile browsers can run the app, but performance varies:
- Android Chrome generally works well.
- iOS Safari works but has stricter camera-permission rules (see Q10).
- Pose detection is CPU/GPU-intensive — on low-end phones you may see reduced frame rates. Future optimization issues are tracked under the
enhancementlabel.
| Requirement | Version | Notes |
|---|---|---|
| Node.js | 18.x or higher | LTS recommended |
| npm or yarn | npm 9+ | Ships with Node 18 |
| Git | any recent version | |
| A modern browser | latest Chrome/Edge/Firefox/Safari | See Q2 |
| Webcam | any USB or built-in | Required for pose features |
| A Firebase project | optional for local dev | Needed only for auth/storage features — see Q6 |
The frontend reads its Firebase configuration from .env.local at the repo root. Start from the template:
cp .env.example .env.localThen fill in the values from your Firebase project (Firebase Console → Project settings → General → Your apps):
VITE_FIREBASE_API_KEY=...
VITE_FIREBASE_AUTH_DOMAIN=your-project.firebaseapp.com
VITE_FIREBASE_PROJECT_ID=your-project-id
VITE_FIREBASE_STORAGE_BUCKET=your-project.appspot.com
VITE_FIREBASE_MESSAGING_SENDER_ID=...
VITE_FIREBASE_APP_ID=...After editing .env.local, restart the dev server so Vite picks up the new values.
⚠️ Never commit.env.local. It's already covered by.gitignore.
Not strictly — the pose-detection and 3D visualization features run without Firebase. You can leave the env values blank and still use the camera + rep-counting flow.
You do need Firebase if you want to:
- Sign in / sign up
- Save workout history
- Test features that read or write to Firestore
SpectraX needs two terminals running side by side.
Terminal 1 — backend (port 3001):
cd server
npm install # first time only
npm run dev # or: npm startTerminal 2 — frontend (port 5173):
npm install # first time only
npm run devThen open http://localhost:5173 in your browser.
📝 The backend port defaults to 3001 and can be overridden via
PORTinserver/.env. The frontend readsVITE_BACKEND_URLfrom the root.env(defaults tohttp://localhost:3001), so you set the port in one place and both sides agree.
A healthy first run looks like this:
- The backend terminal logs that Socket.IO is listening on port
3001. - The frontend terminal shows the Vite ready banner with
Local: http://localhost:5173/. - The browser opens to the SpectraX welcome screen with no red errors in the DevTools Console (
F12). - After granting camera permission, the camera preview appears within ~1–2 seconds.
If any of these fail, jump to the relevant section below.
This error comes from src/services/cameraService.ts when getUserMedia is rejected. Re-enable camera access for http://localhost:5173:
| Browser | Steps |
|---|---|
| Chrome | Click the 🔒/ℹ️ icon left of the URL → Site settings → Camera → Allow. Or visit chrome://settings/content/camera. |
| Edge | 🔒 icon → Permissions for this site → Camera → Allow. Or edge://settings/content/camera. |
| Firefox | 🔒 icon → Connection secure → More information → Permissions → Camera → uncheck Block. Or about:preferences#privacy. |
| Safari | Safari → Settings → Websites → Camera → set localhost to Allow. |
After changing the setting, reload the page (Ctrl/Cmd + R).
Safari and iOS have stricter rules than Chromium-based browsers:
- The page must be served over HTTPS — or be exactly
http://localhost/http://127.0.0.1. Custom hostnames over HTTP will be silently blocked. - iOS Safari requires the camera request to happen inside a user gesture (a click or tap). If you reload via JS or navigate programmatically, the prompt may not appear.
- Installed PWA contexts sometimes restrict camera access further.
If you're testing on a phone against your dev machine, see Q11.
All major browsers require HTTPS for getUserMedia outside localhost. For deployed environments:
- Use a host with TLS (Vercel, Netlify, Render, etc. — they default to HTTPS).
- For quick external testing, tunnel your local dev server with ngrok, cloudflared, or localtunnel — they expose your
localhost:5173over HTTPS.
Common causes:
- Another tab or app (Zoom, Meet, OBS, virtual-camera drivers) is holding the device.
- OS-level camera privacy is disabled. Check:
- Windows: Settings → Privacy & security → Camera → ensure browser access is on.
- macOS: System Settings → Privacy & Security → Camera → tick your browser.
- A virtual-camera driver (Snap Camera, OBS Virtual Camera) is selected as default — disable it and reload.
MediaPipe Pose is loaded from jsDelivr at runtime — see index.html and src/services/poseService.ts. On the first visit it downloads:
pose.js(the JS wrapper)pose_solution_simd_wasm_bin.wasm/.js(the WebAssembly runtime)pose_landmark_*.tflite/.data(the model weights)
Combined, these are roughly 5–15 MB. After the first load they sit in the browser's HTTP cache, so subsequent reloads are near-instant.
Open DevTools → Network and filter by mediapipe. You should see several requests to cdn.jsdelivr.net/npm/@mediapipe/pose/... returning HTTP 200, including .wasm and .data files.
If those requests are blocked or 4xx/5xx:
- Ad blockers / privacy extensions (uBlock Origin, Brave Shields) sometimes block CDN requests — pause them for
localhost:5173. - Corporate proxies may block jsDelivr — try a different network.
- Content-Security-Policy headers added by a browser extension can block WASM evaluation. Disable extensions and reload.
This is expected auto-recovery wired into src/services/poseService.ts. After ~10 consecutive bad frames, the service reinitializes MediaPipe instead of staying stuck.
If you see it persistently:
- Lower the camera resolution (some integrated webcams struggle at 1280×720).
- Make sure hardware acceleration is on in your browser (
chrome://settings/system→ "Use graphics acceleration"). - Close other tabs that are also using the camera or heavy GPU.
The activity classifier (src/workers/activityWorker.ts) lazy-loads the Xenova/clip-vit-base-patch32 model from Hugging Face. Because env.allowLocalModels = false, the worker must reach huggingface.co over the network.
Common causes of failure:
- Corporate firewalls that block
huggingface.co. - Browser cache corruption — open DevTools → Application → Clear storage and reload.
- Slow first download (model is ~150 MB) — give it a minute before assuming failure.
Not today — both MediaPipe and Transformers.js are loaded from public CDNs by default. Self-hosting the WASM and model files is technically possible (MediaPipe accepts a custom locateFile callback) but is out of scope for the default setup. If you want to help add an offline mode, open a Discussion or issue under the enhancement label.
The lint script runs with --max-warnings 0, so every warning blocks the build. Before committing:
npm run lint -- --fix
npm run lintThe first command auto-fixes anything it can; the second confirms a clean exit. CI runs the same command, so a passing local lint should pass CI.
Either another Vite session is still alive, or another app is using the port. Two options:
# free the port (Windows / macOS / Linux)
npx kill-port 5173
# or start on a different port
npm run dev -- --port 5174If you pick a new port, the camera permission still works because both are on localhost.
Checklist:
- Confirm the backend terminal is still running on
:3001. - The backend has
cors: { origin: '*' }andtransports: ['websocket'](server/index.js) — make sure no corporate proxy is stripping WebSocket upgrades. - Reload the frontend tab after the backend starts.
- If you changed the backend port, update the Socket.io client URL in the frontend code to match.
A new dependency or type change is likely. From the repo root:
npm install
cd server && npm install && cd ..Then restart your editor's TypeScript server (in VS Code: Ctrl/Cmd + Shift + P → "TypeScript: Restart TS Server"). Run npm run build again.
Open DevTools Console. Typical culprits:
- Missing env vars in production build — Vite inlines
VITE_*vars at build time, so make sure.env.local(or.env.production) is populated before runningnpm run build. - Sub-path deployment — if you're serving under
/spectrax/instead of/, setbasein vite.config.ts accordingly and rebuild. - Service worker caching an older build — unregister it from DevTools → Application → Service Workers.
The full path from your webcam to a rep count, in five steps:
- Camera capture — src/services/cameraService.ts calls
navigator.mediaDevices.getUserMediaand attaches the stream to a hidden<video>element at 1280×720 / 30 fps. - Pose inference — src/services/poseService.ts hands each frame to MediaPipe Pose, which returns 33 body landmarks (x, y, z, visibility).
- Angle computation — src/workers/poseWorker.ts runs in a Web Worker so the math doesn't block the UI thread; it computes joint angles (knee, elbow, hip) from the landmarks.
- Activity classification — src/workers/activityWorker.ts uses Transformers.js with a CLIP vision model to detect which exercise is being performed.
- Server sync — the backend (server/index.js) mirrors the same angle math so reps stay consistent across devices that join the same Socket.io session.
This separation (camera → pose → worker → classifier → server) makes it easy to swap any single piece without touching the others — perfect for first contributions.
The full process lives in CONTRIBUTING.md. In short:
- Browse open issues labeled
gssoc-26,good first issue,level1,level2, orlevel3. - Comment on the issue you want, using the standard template:
I would like to work on this issue under GSSoC'26. Please assign it to me.
- Wait for a maintainer to assign the issue to you before writing any code.
- Fork the repo, branch with the right prefix (
feature/,bugfix/,docs/,refactor/, ortest/), and commit with the right prefix (feat:,fix:,docs:,style:,refactor:,test:). - Open a PR that references the issue (e.g.
Fixes #37) and fills in the PR template.
Open a new issue using the appropriate template in .github/ISSUE_TEMPLATE/. Include:
- Steps to reproduce
- Expected vs. actual behavior
- Browser, OS, and any console errors
- For features: the user-facing problem you're trying to solve, not just the implementation idea
Do not start coding until a maintainer assigns the issue to you — unassigned PRs may be closed per the contribution rules.
In order of preference:
- Re-read this FAQ and CONTRIBUTING.md — most setup issues are covered.
- Search the Issues tab — someone may have already hit it.
- Open a Discussion describing your environment and what you've tried.
- Comment on the issue assigned to you so the maintainer sees context.
Be specific: paste the exact error message, your Node version (node -v), your OS, and a screenshot of the DevTools console if it's a runtime problem. Vague "it doesn't work" reports are hard to help with.
- README.md — overview, installation, usage
- CONTRIBUTING.md — branch naming, commit format, PR process
- CODE_OF_CONDUCT.md — community standards
SpectraX — built for GSSoC'26 contributors. Happy hacking! 🚀