Skip to content

1) extraction analysis from first recording 2) integrate mediapipe to… #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
temanisyarat merged 1 commit into from
May 20, 2026
Merged

1) extraction analysis from first recording 2) integrate mediapipe to… #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions docs/adr/adr-006-extraction-analysis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Analisis Ekstraksi Landmark dan Rekomendasi Perekaman

## Temuan Utama

Ekstraksi landmark dari 1600 video (8 signer, 20 gesture) menunjukkan tingkat kegagalan yang tinggi pada deteksi face:

| Landmark | Dimensi | Rata-rata NaN |
|----------|---------|:------------:|
| Pose | 27 | **0.1%** |
| Face | 99 | **84.5%** |
| Hands | 126 | **29.8%** |

Face gagal terdeteksi di sebagian besar frame, bahkan dengan confidence threshold minimal (0.05) dan preprocessing CLAHE.

### Pola Kegagalan Face

| Signer | Face NaN | Hands NaN |
|----------|:--------:|:---------:|
| farras | 44.2% | 14.4% |
| willi | 64.4% | 7.7% |
| ian | 74.8% | 8.3% |
| hani | 98.3% | 49.6% |
| mutia | 97.3% | 67.8% |
| fredi | 97.4% | 9.9% |
| saidah | 99.8% | 60.6% |
| ivan | 100.0% | 20.2% |

Pola ini menunjukkan bahwa akar masalah bukan pada pipeline ekstraksi atau threshold model, melainkan pada **kualitas perekaman**: signer tidak menghadap kamera, pencahayaan buruk, atau wajah tidak terlihat.

### Dampak pada Akurasi Model

Pipeline model sebelumnya menggunakan 252 dimensi input (pose + face + hands). Dengan 84.5% face dan 29.8% hands NaN, model hanya belajar dari ~27 dimensi data bersih. ~~Hasil akhir: akurasi ~10-16% pada test set (setara random untuk 20 kelas).~~

## Keputusan

1. **Drop face dari pipeline** — face tidak diekstrak lagi di `extractor.py` dan tidak digunakan di model. Input dimensi turun dari 252 ke 153 (pose 27 + hands 126).
2. **Pipeline model sudah diperbaiki** — arsitektur GRU 1 layer (96 hidden), label smoothing, ReduceLROnPlateau, gradient clipping.

## Rekomendasi Perekaman ke Depan

### Posisi Kamera
- **Kamera setara mata** — tidak dari atas/bawah agar wajah terekam frontal
- **Frame upper body** — kepala + kedua tangan + dada harus selalu dalam frame
- Resolusi 720p sudah cukup baik — pertahankan

### Pencahayaan
- **Cahaya depan (frontal)** — hindari backlight (jendela di belakang signer)
- Gunakan **diffuse light** — hindari bayangan keras di wajah
- Standar: mean brightness ~150-200 dengan std dev >40 (cek dengan histogram)

### Instruksi untuk Signer
- **Hadap kamera** — jangan melihat ke tangan saat memberi gesture
- **Tangan di frame** — jangan gesture terlalu rendah atau ke samping
- **Pakaian kontras** dengan background — hindari warna kulit/skin tone di background untuk deteksi hands

### Sesi Perekaman
- 1 signer per sesi — tidak boros biaya transport/harian
- Rekam per gesture dalam 1 take kontinu — lebih mudah daripada per video pendek
- Validasi cepat: minta signer review 2-3 video untuk memastikan landmark terdeteksi

### QC Checklist (Sebelum Simpan)
- [ ] Pose terdeteksi di semua frame (<1% NaN)
- [ ] Wajah terlihat jelas di >80% frame
- [ ] Kedua tangan terdeteksi di >90% frame
- [ ] Background tidak berantakan (deteksi tangan lebih baik dengan BG polos)
123 changes: 123 additions & 0 deletions docs/spec/spec-005-integrate-mediapipe-flutter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# spec-005: Integrate MediaPipe + Flutter (Hand Landmarker)

## Overview

Integrate MediaPipe Hand Landmarker into the Teman Isyarat Flutter app
using a native Android `PlatformView` for real-time camera + hand
tracking, bridged via `AndroidView` widget in Flutter.

## Architecture

```
Flutter (Dart)
┌───────────────┐ ┌──────────────────────────┐
│ TranslatePage │ │ AndroidView │
│ (Dart UI) │──▶│ (PlatformView) │
│ + result box │ │ ┌────────────────────┐ │
│ + nav │ │ │ MethodChannel │ │
└───────────────┘ │ └──────┬─────────────┘ │
└────────┼─────────────────┘
│
Native Android (Kotlin) │
┌────────────────────────────▼──────────────────┐
│ HandLandmarkerView (PlatformView) │
│ ┌────────────────┐ ┌────────────────────┐ │
│ │ CameraX │ │ HandLandmarker │ │
│ │ (PreviewView) │ │ OverlayView │ │
│ └───────┬────────┘ │ (canvas skeleton) │ │
│ │ └────────────────────┘ │
│ ┌───────▼──────────────────────────────────┐ │
│ │ HandLandmarkerHelper (MediaPipe Tasks) │ │
│ │ - init / setup │ │
│ │ - detectLiveStream (LIVE_STREAM mode) │ │
│ │ - callback -> onResults / onError │ │
│ └──────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────┐ │
│ │ GestureClassifier (future) │ │
│ │ - receives HandLandmarkerResult │ │
│ │ - returns gesture label + confidence │ │
│ └──────────────────────────────────────────┘ │
└───────────────────────────────────────────────┘
```

## Files to Create

| # | File | Purpose |
| --- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| 1 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerHelper.kt` | MediaPipe Tasks wrapper - init, config, detectLiveStream, result/error listeners |
| 2 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerView.kt` | PlatformView - CameraX + PreviewView + overlay + lifecycle |
| 3 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerOverlay.kt` | Custom View - draws hand skeleton (21 landmarks + HAND_CONNECTIONS) |
| 4 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerPlugin.kt` | PlatformViewFactory + MethodChannel handler |
| 5 | `lib/pages/translate_page.dart` | TranslatePage refactored out of main.dart, uses AndroidView |

## Files to Modify

| # | File | Changes |
|---|------|---------|
| 1 | `android/app/build.gradle.kts` | Add `com.google.mediapipe:tasks-vision:0.10.29`, CameraX deps, set `minSdk = 24` |
| 2 | `android/app/src/main/AndroidManifest.xml` | Add `<uses-permission android:name="android.permission.CAMERA" />` |
| 3 | `lib/main.dart` | Replace TranslatePage placeholder with real AndroidView widget |

## Assets

| # | File | Source |
|---|------|--------|
| 1 | `android/app/src/main/assets/hand_landmarker.task` | `https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task` |

## MethodChannel API

### Dart -> Native

| Call | Args | Returns | Notes |
|------|------|---------|-------|
| `startCamera` | - | `bool` | Initializes CameraX + MediaPipe |
| `stopCamera` | - | `bool` | Releases resources |
| `switchCamera` | - | `bool` | Front <-> back |
| `updateSettings` | `{maxHands: int, detectionConfidence: float, trackingConfidence: float, delegate: int}` | `bool` | 0=CPU, 1=GPU |

### Native -> Dart (via callback channel)

| Event | Data | Description |
|-------|------|-------------|
| `onGestureResult` | `{gesture: String, confidence: float}` | Gesture classification (future) |
| `onError` | `{message: String}` | Error reporting |
| `onLandmarks` | `{landmarks: [[x,y,z]]}` | Raw landmark positions |

## Gradle Dependencies

```kotlin
// MediaPipe Tasks Vision
implementation("com.google.mediapipe:tasks-vision:0.10.29")

// CameraX
val cameraxVersion = "1.4.2"
implementation("androidx.camera:camera-core:$cameraxVersion")
implementation("androidx.camera:camera-camera2:$cameraxVersion")
implementation("androidx.camera:camera-lifecycle:$cameraxVersion")
implementation("androidx.camera:camera-view:$cameraxVersion")
```

## Implementation Order

1. Add Gradle dependencies + CAMERA permission
2. Download `hand_landmarker.task` to `android/app/src/main/assets/`
3. Create `HandLandmarkerHelper.kt` - adapted from Google's example
4. Create `HandLandmarkerOverlay.kt` - Canvas landmark skeleton
5. Create `HandLandmarkerView.kt` - PlatformView with CameraX
6. Create `HandLandmarkerPlugin.kt` - Factory + MethodChannel
7. Register plugin in MainActivity
8. Refactor TranslatePage + AndroidView into Dart
9. Build & verify

## Future: Gesture Classification

`HandLandmarkerHelper` includes a pluggable callback:

```kotlin
fun interface GestureClassifier {
fun classify(landmarks: HandLandmarkerResult): Pair<String, Float>
}
```

When the classifier is set, `onResults` runs it and sends `onGestureResult` to Dart.
Classifier implementation (e.g. TF Lite model) can be added later without touching CameraX or MediaPipe pipeline.
Loading