Skip to content

Commit 65ffab9

Browse files
temanisyarattemanisyarat
authored
Merge pull request #40 from williamu04/adr6/spec5
1) extraction analysis from first recording 2) integrate mediapipe to…
2 parents 79b2c36 + 392dae6 commit 65ffab9

2 files changed

Lines changed: 188 additions & 0 deletions

File tree

‎docs/adr/adr-006-extraction-analysis.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Analisis Ekstraksi Landmark dan Rekomendasi Perekaman
2+
3+
## Temuan Utama
4+
5+
Ekstraksi landmark dari 1600 video (8 signer, 20 gesture) menunjukkan tingkat kegagalan yang tinggi pada deteksi face:
6+
7+
| Landmark | Dimensi | Rata-rata NaN |
8+
|----------|---------|:------------:|
9+
| Pose | 27 | **0.1%** |
10+
| Face | 99 | **84.5%** |
11+
| Hands | 126 | **29.8%** |
12+
13+
Face gagal terdeteksi di sebagian besar frame, bahkan dengan confidence threshold minimal (0.05) dan preprocessing CLAHE.
14+
15+
### Pola Kegagalan Face
16+
17+
| Signer | Face NaN | Hands NaN |
18+
|----------|:--------:|:---------:|
19+
| farras | 44.2% | 14.4% |
20+
| willi | 64.4% | 7.7% |
21+
| ian | 74.8% | 8.3% |
22+
| hani | 98.3% | 49.6% |
23+
| mutia | 97.3% | 67.8% |
24+
| fredi | 97.4% | 9.9% |
25+
| saidah | 99.8% | 60.6% |
26+
| ivan | 100.0% | 20.2% |
27+
28+
Pola ini menunjukkan bahwa akar masalah bukan pada pipeline ekstraksi atau threshold model, melainkan pada **kualitas perekaman**: signer tidak menghadap kamera, pencahayaan buruk, atau wajah tidak terlihat.
29+
30+
### Dampak pada Akurasi Model
31+
32+
Pipeline model sebelumnya menggunakan 252 dimensi input (pose + face + hands). Dengan 84.5% face dan 29.8% hands NaN, model hanya belajar dari ~27 dimensi data bersih. ~~Hasil akhir: akurasi ~10-16% pada test set (setara random untuk 20 kelas).~~
33+
34+
## Keputusan
35+
36+
1. **Drop face dari pipeline** — face tidak diekstrak lagi di `extractor.py` dan tidak digunakan di model. Input dimensi turun dari 252 ke 153 (pose 27 + hands 126).
37+
2. **Pipeline model sudah diperbaiki** — arsitektur GRU 1 layer (96 hidden), label smoothing, ReduceLROnPlateau, gradient clipping.
38+
39+
## Rekomendasi Perekaman ke Depan
40+
41+
### Posisi Kamera
42+
- **Kamera setara mata** — tidak dari atas/bawah agar wajah terekam frontal
43+
- **Frame upper body** — kepala + kedua tangan + dada harus selalu dalam frame
44+
- Resolusi 720p sudah cukup baik — pertahankan
45+
46+
### Pencahayaan
47+
- **Cahaya depan (frontal)** — hindari backlight (jendela di belakang signer)
48+
- Gunakan **diffuse light** — hindari bayangan keras di wajah
49+
- Standar: mean brightness ~150-200 dengan std dev >40 (cek dengan histogram)
50+
51+
### Instruksi untuk Signer
52+
- **Hadap kamera** — jangan melihat ke tangan saat memberi gesture
53+
- **Tangan di frame** — jangan gesture terlalu rendah atau ke samping
54+
- **Pakaian kontras** dengan background — hindari warna kulit/skin tone di background untuk deteksi hands
55+
56+
### Sesi Perekaman
57+
- 1 signer per sesi — tidak boros biaya transport/harian
58+
- Rekam per gesture dalam 1 take kontinu — lebih mudah daripada per video pendek
59+
- Validasi cepat: minta signer review 2-3 video untuk memastikan landmark terdeteksi
60+
61+
### QC Checklist (Sebelum Simpan)
62+
- [ ] Pose terdeteksi di semua frame (<1% NaN)
63+
- [ ] Wajah terlihat jelas di >80% frame
64+
- [ ] Kedua tangan terdeteksi di >90% frame
65+
- [ ] Background tidak berantakan (deteksi tangan lebih baik dengan BG polos)

‎docs/spec/spec-005-integrate-mediapipe-flutter.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# spec-005: Integrate MediaPipe + Flutter (Hand Landmarker)
2+
3+
## Overview
4+
5+
Integrate MediaPipe Hand Landmarker into the Teman Isyarat Flutter app
6+
using a native Android `PlatformView` for real-time camera + hand
7+
tracking, bridged via `AndroidView` widget in Flutter.
8+
9+
## Architecture
10+
11+
```
12+
Flutter (Dart)
13+
┌───────────────┐ ┌──────────────────────────┐
14+
│ TranslatePage │ │ AndroidView │
15+
│ (Dart UI) │──▶│ (PlatformView) │
16+
│ + result box │ │ ┌────────────────────┐ │
17+
│ + nav │ │ │ MethodChannel │ │
18+
└───────────────┘ │ └──────┬─────────────┘ │
19+
└────────┼─────────────────┘
20+
│
21+
Native Android (Kotlin) │
22+
┌────────────────────────────▼──────────────────┐
23+
│ HandLandmarkerView (PlatformView) │
24+
│ ┌────────────────┐ ┌────────────────────┐ │
25+
│ │ CameraX │ │ HandLandmarker │ │
26+
│ │ (PreviewView) │ │ OverlayView │ │
27+
│ └───────┬────────┘ │ (canvas skeleton) │ │
28+
│ │ └────────────────────┘ │
29+
│ ┌───────▼──────────────────────────────────┐ │
30+
│ │ HandLandmarkerHelper (MediaPipe Tasks) │ │
31+
│ │ - init / setup │ │
32+
│ │ - detectLiveStream (LIVE_STREAM mode) │ │
33+
│ │ - callback -> onResults / onError │ │
34+
│ └──────────────────────────────────────────┘ │
35+
│ ┌──────────────────────────────────────────┐ │
36+
│ │ GestureClassifier (future) │ │
37+
│ │ - receives HandLandmarkerResult │ │
38+
│ │ - returns gesture label + confidence │ │
39+
│ └──────────────────────────────────────────┘ │
40+
└───────────────────────────────────────────────┘
41+
```
42+
43+
## Files to Create
44+
45+
| # | File | Purpose |
46+
| --- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
47+
| 1 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerHelper.kt` | MediaPipe Tasks wrapper - init, config, detectLiveStream, result/error listeners |
48+
| 2 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerView.kt` | PlatformView - CameraX + PreviewView + overlay + lifecycle |
49+
| 3 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerOverlay.kt` | Custom View - draws hand skeleton (21 landmarks + HAND_CONNECTIONS) |
50+
| 4 | `android/app/src/main/java/com/example/android/handlandmarker/HandLandmarkerPlugin.kt` | PlatformViewFactory + MethodChannel handler |
51+
| 5 | `lib/pages/translate_page.dart` | TranslatePage refactored out of main.dart, uses AndroidView |
52+
53+
## Files to Modify
54+
55+
| # | File | Changes |
56+
|---|------|---------|
57+
| 1 | `android/app/build.gradle.kts` | Add `com.google.mediapipe:tasks-vision:0.10.29`, CameraX deps, set `minSdk = 24` |
58+
| 2 | `android/app/src/main/AndroidManifest.xml` | Add `<uses-permission android:name="android.permission.CAMERA" />` |
59+
| 3 | `lib/main.dart` | Replace TranslatePage placeholder with real AndroidView widget |
60+
61+
## Assets
62+
63+
| # | File | Source |
64+
|---|------|--------|
65+
| 1 | `android/app/src/main/assets/hand_landmarker.task` | `https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task` |
66+
67+
## MethodChannel API
68+
69+
### Dart -> Native
70+
71+
| Call | Args | Returns | Notes |
72+
|------|------|---------|-------|
73+
| `startCamera` | - | `bool` | Initializes CameraX + MediaPipe |
74+
| `stopCamera` | - | `bool` | Releases resources |
75+
| `switchCamera` | - | `bool` | Front <-> back |
76+
| `updateSettings` | `{maxHands: int, detectionConfidence: float, trackingConfidence: float, delegate: int}` | `bool` | 0=CPU, 1=GPU |
77+
78+
### Native -> Dart (via callback channel)
79+
80+
| Event | Data | Description |
81+
|-------|------|-------------|
82+
| `onGestureResult` | `{gesture: String, confidence: float}` | Gesture classification (future) |
83+
| `onError` | `{message: String}` | Error reporting |
84+
| `onLandmarks` | `{landmarks: [[x,y,z]]}` | Raw landmark positions |
85+
86+
## Gradle Dependencies
87+
88+
```kotlin
89+
// MediaPipe Tasks Vision
90+
implementation("com.google.mediapipe:tasks-vision:0.10.29")
91+
92+
// CameraX
93+
val cameraxVersion = "1.4.2"
94+
implementation("androidx.camera:camera-core:$cameraxVersion")
95+
implementation("androidx.camera:camera-camera2:$cameraxVersion")
96+
implementation("androidx.camera:camera-lifecycle:$cameraxVersion")
97+
implementation("androidx.camera:camera-view:$cameraxVersion")
98+
```
99+
100+
## Implementation Order
101+
102+
1. Add Gradle dependencies + CAMERA permission
103+
2. Download `hand_landmarker.task` to `android/app/src/main/assets/`
104+
3. Create `HandLandmarkerHelper.kt` - adapted from Google's example
105+
4. Create `HandLandmarkerOverlay.kt` - Canvas landmark skeleton
106+
5. Create `HandLandmarkerView.kt` - PlatformView with CameraX
107+
6. Create `HandLandmarkerPlugin.kt` - Factory + MethodChannel
108+
7. Register plugin in MainActivity
109+
8. Refactor TranslatePage + AndroidView into Dart
110+
9. Build & verify
111+
112+
## Future: Gesture Classification
113+
114+
`HandLandmarkerHelper` includes a pluggable callback:
115+
116+
```kotlin
117+
fun interface GestureClassifier {
118+
fun classify(landmarks: HandLandmarkerResult): Pair<String, Float>
119+
}
120+
```
121+
122+
When the classifier is set, `onResults` runs it and sends `onGestureResult` to Dart.
123+
Classifier implementation (e.g. TF Lite model) can be added later without touching CameraX or MediaPipe pipeline.

0 commit comments

Comments
 (0)