eye-gaze

an eye gaze data gathering , training , and inference toolset

Setup

Setup for data gathering and pre processing Conda is reccomended for ease of use

conda create -n py310-data python=3.10
conda activate py310
pip install opencv-python numpy mediapipe pyautogui pandas tqdm pandas matplotlib

Setup for training

conda create -n py312-train python=3.12
conda activate py312
pip install opencv-python tensorflow[and-cuda] tqdm matplotlib pandas numpy

latest model is provided BESTOVERALL.keras

Configuration

main configuration is at the class SETTINGS in src\utilities\__init__.py per file configurations and settings are also available and not all settings are in class SETTINGS

How to use ?

Training

follow from a to f

Calibration

• 1080p 30fps videos are reccomended for best results

1. Configure the scripts with propper settings
2. Gather data with b_get_data_template.py
3. Split csv with t_csv_splitter.py
4. Preprocess data with c_preprocess.py
5. Remove the iris_distances.csv file
6. Calibrate with f_train_calibration_external.py

Files Explanation

Data Gathering and PreProcessing

These files should use py310-data enviroment due to mediapipes availability

a_calibrate_stereo.py : Calibrate your stereo setup stereo calibration info : https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html stereo calibration squares : https://github.com/opencv/opencv/blob/4.x/doc/pattern.png

before calibration please ensure your squares mm sizes and update the square size and count info in script
at calibration use some kind of support to hold the square pattern more straight
after calibration you can check calibration outputs and see if they are true or not

b_get_data_template.py : Gather webcam data, shows videos & dots class ScenePlayer can be costimized to your needs to gather data just create a scene list as your needs Scenes also can be customized with diffrent modes

"G"     : Previous Scene
"H"     : Next Scene
"Space" : Space Action || Ready to progress in the scene itself
"R"     : Toggle Animations Off || Start the scene on movement scenes
"T"     : Toggle Animations On
"Q"     : Close the application

reccomended mininmum people for [train, validation, test] = [4, 1, 1]
per person it generates 13.5GBs~ per cameara for default movement scene & 2 minute video

c_preprocess.py: Preprocess data logic it includes pre processes the data logic it doesnt do anything itself except if you modify it if you have more than one people you can use threaded version to auto run for all people in a dataset

preprocesses the full frame images into eye crops and feature vectors per person

please use `t_csv_splitter.py` on all persons before this 

structures of feature vectors

```python
    # Structure of Monocular Vector :
    # 1. selected_landmarks_3d (len(SELECTED_TRAX_INDICES))
    # 2. Head pose (3 vals) 
    # 3. EAR (4 vals) 
    # 4. IOD & HW (2 vals) 

    # Structure of Binocular Vector :
    # 1. selected_landmarks_3d (len(SELECTED_TRAX_INDICES)*3)
    # 2. 2D metrics L/R (4 vals) 
    # 3. 2D pose L (3 vals) 
    # 4. 2D pose R (3 vals)
    # 5. 3D pose (3 vals) 
    # 6. 3D metrics (2 vals) 
    # 7. 3D distances (2 vals)
```

c_preprocess_all_threaded.py : Calls preprocess for all persons in a dataset please use t_csv_splitter.py on all persons before this

d_combine_data.py : Combines all persons in a dataset in to "ALL_" files it is not required for calibration just for training

Training , Calibration and Inference

These files should use py312-train enviroment due to compatibility with Google Collab

e_train_general.py : Trains a general model , tests it on held out person , outputs video if video is available results show in folder "/experiment/..." as model.keras and result.txt

f_train_calibration.py : Calibration trains the heldout person from the model to evaluate the model for all calibration files model settings should match the trained general model

f_train_calibration_avg.py : Uses averages of both eye outputs to generate the output for inference result

f_train_calibration_external.py : Calibration trains the selected model on your chosen new person this is the script you should use if you want to get results from a calibration select your video , person to calibrate and run

currently this requires diffrent names for some files please rename them after preprocessing and please delete the "iris_distances.csv"

currently there is a desync between eye captures and current frames

t_csv_splitter.py : Splits frames.csv into mouse_frames.csv and modes.csv for backwards compatibility

t_coverage_map.py : Creates coverage map from dataset

Utilities

utilites\__init__.py: Holds the constant values and the class SETTINGS

utilities\get_data\... : Holds data gathering related files camera.py : Handles camera logic of reading info and saving it uses threads for each camera movement.py : Handles the logic of movements of the dots video.py : Handles the logic of Video playing with vlc, audio is also supported

utilities\get_data\scene\... : Holds scene class files
scene_player : Scene Player class plays the scenes and handles logic base.py: Base Scene class info_scene.py: Info scene shows info lines movement_scene.py: Visualises and handles Movement Class logic to show dots video_scene.py: Shows selected video

utilities\preprocess\... : Holds preprocess helper functions binocular_features.py : Handles helpers for binocular feature extraction extract_features.py : Extract features function landmarks.py : Constant landmarks preprocess_helpers.py : other default preprocess helpers

Be aware before use

• this model is trained for 1080p 27" screen with center bottom camera placement

• many webcams drop their fps in half when enviroment is too dark

• sometimes webcams may not use 30fps

• larger MP sensors result in better results

• glasses result in more errors

• in windows you may not be getting 30 fps

Known Issues

• sync for video playing is not working
• must remove the iris_distances.csv for calibration purposes
• some bugs in the menu when reaching ends of the scenes
• use leftmost screen and also use 1080 for free roam capturing it doesnt handle it there is no logic also just calib points support HighRes yet
• the logitech C505s i use shows mismatch erros but output seems working , current issue is i wouldnt be able to disable error messages from those cameras

Paper

read more about the project paper

Results

latest model: `BESTOVERALL.keras`
before : calibration 163px , 3,04 degree error 
after  : calibration 87px , 1,62 degree error 

Licence

MIT