Skip to content

IMPLEMENTATION_PLAN.md

Latest commit

 

History

History
72 lines (54 loc) · 2.26 KB

IMPLEMENTATION_PLAN.md

File metadata and controls

72 lines (54 loc) · 2.26 KB
Raw

Gesture-Based 3D Design and Physics Validation

Problem Statement

Traditional 2D input devices are not natural for interacting with 3D environments. This project replaces mouse-based control with temporal hand-motion understanding and validates whether manipulated designs remain physically plausible.

Research Positioning

  • Computer vision for robust hand landmark extraction.
  • Deep learning for temporal intent recognition.
  • Human-computer interaction for natural 3D manipulation.
  • Lightweight design validation inspired by physics-aware optimization research.

Objective

Build an interactive system where users can create, move, rotate, scale, join, and delete 3D objects through hand signs, then receive instant structural feedback.

Technical Stack

  • Hand tracking: MediaPipe Hands.
  • Temporal model: LSTM or Transformer.
  • Rendering and interaction: Three.js.
  • API and inference layer: FastAPI.
  • Numeric processing: NumPy.

Pipeline

  1. Webcam stream capture.
  2. Hand landmark extraction (21 points per frame).
  3. Sequence buffering over a temporal window.
  4. Gesture intent classification.
  5. 3D action mapping.
  6. Physics heuristic validation and feedback.

Milestones (7 Weeks)

  • Week 1-2:
    • Implement camera capture and landmarks.
    • Display stable tracking.
  • Week 3-4:
    • Build dataset and train baseline LSTM.
    • Evaluate model with confusion matrix.
  • Week 5:
    • Integrate model output with Three.js actions.
    • Demonstrate create, move, rotate, scale, delete.
  • Week 6:
    • Add physics scoring and warning overlay.
    • Validate behavior on predefined test scenes.
  • Week 7:
    • UX polish, latency reduction, final presentation and report.

MVP Constraints

  • Single hand for first version.
  • Primitive shapes only.
  • Heuristic physics in place of full finite element simulation.

Demo Script

  1. Show real-time hand tracking.
  2. Create an object using a gesture.
  3. Grab and move object in 3D scene.
  4. Rotate and scale object.
  5. Trigger physics check and show stability/stress metrics.
  6. Demonstrate warning and suggestion output.

Success Criteria

  • Gesture intent recognition accuracy above baseline threshold.
  • End-to-end interaction latency acceptable for real-time use.
  • Physics feedback correctly flags intentionally unstable examples.