Proposal: Explore modern gaze correction architectures (2023-2025 research advances)

[circlenaut]

Summary

The current implementation provides a solid, well-architected foundation for real-time gaze correction. The research field has evolved significantly since the original DeepWarp paper (2019), with newer methods achieving substantially improved visual quality through 3D-aware approaches. This issue proposes exploring integration of modern architectures while preserving the repo's strong infrastructure.

---

What This Repo Does Well

Before discussing changes, I want to acknowledge the strengths of the current design:

• Clean abstraction: FacePredictor interface allows swapping backends (dlib ↔ MediaPipe) without touching correction logic
• Modular architecture: Face detection, gaze correction, and display are properly decoupled
• Production-ready scaffolding: Calibration UI, YAML config, SQLite settings persistence, camera handling
• Documentation: Thorough README with architecture diagrams and data flow explanations

This infrastructure is genuinely valuable and worth preserving.

---

The Quality Gap

The current model operates on 48×64 eye patches using 2D warping. This approach has inherent limitations:

• Visible seams where corrected eye patches meet the original frame
• Limited ability to handle large gaze angle corrections
• Warping artifacts become noticeable under close inspection

Users can tell when gaze correction is active, which undermines the goal of natural-looking eye contact.

---

Research Landscape (2019-2025)

The field has progressed through several generations:

Year	Method	Approach	Key Advancement
2019	DeepWarp (current)	2D patch warping	Real-time capable
2020	ST-ED, interpGaze	Encoder-decoder, GANs	Better disentanglement
2023	GazeNeRF (CVPR)	Neural Radiance Fields	3D-aware, rigid rotation
2025	GazeGaussian (ICCV Highlight)	3D Gaussian Splatting	Real-time 3D, state-of-the-art quality

GazeGaussian is particularly interesting because 3DGS was designed for real-time rendering, potentially making it viable for live video correction.

Repository: https://github.com/ucwxb/GazeGaussian
Paper: ICCV 2025 (Highlight)
Pretrained weights: Available on Hugging Face

---

Proposed Path Forward

Rather than replacing the current implementation, I'd suggest an additive approach:

1. Evaluate GazeGaussian feasibility

   • Test inference speed on consumer hardware
   • Determine input requirements (full face vs. eye patches, head pose, etc.)
   • Assess whether FaceData interface needs extension

2. Implement as V2 backend

   • Create GazeCorrectorV2 alongside existing implementation
   • Allow runtime selection between v1 (fast/lightweight) and v2 (quality)
   • Maintain backward compatibility

3. Interface considerations

   • GazeGaussian may require full-face input rather than eye patches
   • May need head pose estimation (could leverage MediaPipe's existing capabilities)
   • FaceData dataclass might need additional fields

4. Dependency management

   • GazeGaussian is PyTorch + CUDA
   • Could keep TF for v1, PyTorch for v2 (optional dependency)
   • Or migrate v1 to PyTorch for consistency

---

Open Questions

• Is there interest in pursuing higher-quality correction at the cost of increased compute?
• Would a "quality tier" system (fast vs. quality mode) be valuable?
• Are there constraints (hardware targets, dependency preferences) that should guide architecture choices?

---

References

• GazeGaussian: https://github.com/ucwxb/GazeGaussian
• GazeNeRF: https://github.com/AlessandroRuzzi/GazeNeRF
• STED-gaze: https://github.com/zhengyuf/STED-gaze
• Original DeepWarp paper: Hsu et al., "Look at Me! Correcting Eye Gaze in Live Video Communication" (ACM TOMM 2019)

---

Happy to help scope this further or prototype the integration if there's interest.

[huxinhai]

I have a question. I see that NVIDIA Maxine's eye-tracking effect is very powerful, but it seems to be unusable on Mac. Demonstration

[huxinhai]

This implementation plan seems to be even better than GazeGaussian. @circlenaut RTGaze