CAIE NYP Batch 4 — Technical Assessment (ElderGuard Activity Level)

Candidate

• Full Name (NRIC): Tan Kay Kit
• Email (same as application): 233809P@mymail.nyp.edu.sg

---

Project Overview

This repository contains:

1. EDA notebook (eda.ipynb): data understanding, data quality checks, imputation logic, and insights.
2. ML pipeline (src/pipeline.py): loads data from SQLite, performs cleaning + imputation + feature engineering + preprocessing, trains 3+ models, evaluates performance, and saves outputs.

Goal: predict activity_level (low_activity, moderate_activity, high_activity) using environmental sensor + context features.

---

Folder Structure

.
├── .github
│   └── workflows
│       └── github-actions.yml 
├── src/
│   ├── config.py                 # Config: db path, test size, random seed, output dirs
│   ├── data.py                   # SQLite helpers (list_tables, load_table)
│   ├── preprocessing.py          # basic_cleaning(), split_xy()
│   ├── imputation.py             # grouped_regression_impute, grouped_mode_impute
│   ├── feature_engineering.py    # add_engineered_features()
│   ├── features.py               # build_preprocessor() (encode/scale)
│   ├── training.py               # train_all_models() (3+ models + tuning)
│   ├── evaluate.py               # eval_report() (macro F1, report, confusion matrix)
│   ├── importance.py             # top_permutation_importance()
│   └── pipeline.py               # main pipeline entrypoint
├── eda.ipynb                     # EDA (required)
├── requirements.txt              # Python dependencies
├── run.sh                        # One-command run from repo root
├── outputs/                      # metrics.csv, best_params.json, reports, feature_importance.json
├── models/                       # saved model files + preprocessor + label encoder
└── .gitignore                    # It is to prevent the files or folders to push to github repo

---

How to Run

1. Install dependencies pip install -r requirements.txt
2. Ensure the database exists Put the SQLite DB at: data/gas_monitoring.db
3. Run the pipeline bash run.sh

How to Modify Parameters

test-size: test fraction (default: 0.2) seed: random seed (default: 42)

Hyperparameter tuning (RandomizedSearchCV)

I fine-tuned the models using RandomizedSearchCV to efficiently search the hyperparameter space without needing to exhaustively try every combination.

• Why RandomizedSearchCV: faster than GridSearchCV for large search spaces while still finding strong parameter sets.
• Scoring metric: f1_macro (chosen due to class imbalance).
• Cross-validation: performed during tuning to estimate generalisation performance.
• Where to edit: update the search spaces (rf_param_dist, et_param_dist, xgb_param_dist) in src/training.py.
• Outputs saved: best hyperparameters are written to outputs/best_params.json, and tuned reports are saved under outputs/*_report.txt.

Model tuning is controlled by the parameter search space in src/training.py.

To tune the Random Forest model, edit rf_param_dist: model__n_estimators: number of trees (more trees usually improves stability but increases time). model__max_depth: maximum depth per tree (deeper = more complex, risk of overfitting). model__min_samples_leaf: minimum samples in a leaf (higher values reduce overfitting). model__min_samples_split: minimum samples needed to split (higher values reduce overfitting). model__max_features: number of features considered per split (controls randomness + generalisation). model__bootstrap: whether to sample with replacement (affects diversity of trees). model__class_weight: handles class imbalance (useful because high_activity is the minority class).

Extra Trees hyperparameter tuning (edit in src/training.py)

To tune the ExtraTrees model, edit the et_param_dist search space: model__n_estimators: number of trees (more trees → more stable but slower). model__max_depth: maximum depth of each tree (deeper → more complex, can overfit). model__min_samples_leaf: minimum samples in each leaf (higher → smoother model, less overfit). model__min_samples_split: minimum samples to split a node (higher → less overfit). model__max_features: number of features considered at each split (controls randomness/generalisation). model__bootstrap: whether samples are drawn with replacement (changes tree diversity). model__class_weight: handles class imbalance (important because high_activity is minority).

XGBoost hyperparameter tuning (edit in src/training.py)

To tune the XGBoost model, edit the xgb_param_dist search space: model__n_estimators: number of boosting rounds (more rounds → higher capacity, but slower). model__learning_rate: step size per round (smaller → needs more trees, often generalises better). model__max_depth: tree depth (deeper → can model complex patterns but overfits easier). model__subsample: fraction of rows used per tree (lower → more randomness, less overfitting). model__colsample_bytree: fraction of features used per tree (lower → less overfitting). model__min_child_weight: minimum weight in a leaf (higher → more conservative, reduces overfitting). model__gamma: minimum loss reduction to split (higher → fewer splits, more conservative). model__reg_lambda (L2) and model__reg_alpha (L1): regularisation terms (help control overfitting).

Pipeline Flow (Step-by-step)

Load data (SQLite)

Load table from data/gas_monitoring.db.

Basic cleaning

Remove duplicates. Standardize labels (trim/case/snake_case). Convert suspicious/invalid sensor values to missing (NaN) using conservative rules (contamination-safe).

Staged imputation

Group-based using (session_id, time_of_day): humidity ← regression using co2_electro_chemical_sensor (clip 0–100) metal_oxide_sensor_unit2 ← regression using metal_oxide_sensor_unit4 co2_infrared_sensor ← regression using co2_electro_chemical_sensor (clip to reasonable bounds) co_gas_sensor ← grouped mode impute (discrete/ordinal-like) ambient_light_level ← grouped mode impute (categorical)

Feature engineering

CO₂ consistency: co2_diff, co2_ratio Metal oxide aggregates: mean/std/min/max/range across units 1–4 Comfort features: dew point (temperature + humidity) Drop non-predictive / leakage-prone columns Drop session_id (ID leakage risk). Drop *_impute_stage and #_missing columns (tracking only, not generalizable).

Features

One-hot encode categorical features.

Train 3 models

Random Forest Extra Trees XGBoost

Evaluate & save artifacts

Classification report + confusion matrix per model Save metrics.csv, best_params.json, model files, preprocessor, label encoder Permutation feature importance

Key EDA Findings (Short Summary)

Missingness: humidity, metal_oxide_sensor_unit2, ambient_light_level, co_gas_sensor had non-trivial missing values → used structured imputation rather than dropping features.

Duplicates: 171 duplicates removed to avoid biased distributions and inflated performance.

Outliers / contamination risk: invalid humidity (>100 or <0), suspicious temperature values (Kelvin-like), extreme/near-zero CO₂/CO readings → applied conservative validity rules and treated invalid values as missing.

Target imbalance: low_activity dominates; high_activity is minority → accuracy can be misleading → main metric is Macro F1.

Most predictive signals (from bivariate analysis): metal oxide sensors (especially Unit 4/2/3), co2_electro_chemical_sensor, and co_gas_sensor categories. Detailed EDA is in eda.ipynb.

Feature Processing Table

Feature	Type	Processing
temperature	Numeric	validity checks, scaling
humidity	Numeric	invalid→NaN, regression impute via CO₂ electrochemical, clip 0–100, scaling
co2_infrared_sensor	Numeric	invalid→NaN, regression impute via CO₂ electrochemical, clip range, scaling
co2_electro_chemical_sensor	Numeric	validity checks, scaling
metal_oxide_sensor_unit1/3/4	Numeric	validity checks (non-negative), scaling
metal_oxide_sensor_unit2	Numeric	regression impute via Unit 4, scaling
co_gas_sensor	Discrete/Ordinal	grouped mode impute; ordinal or one-hot (validated)
time_of_day	Categorical	one-hot encode
hvac_operation_mode	Categorical	one-hot encode
ambient_light_level	Categorical	grouped mode impute; one-hot encode (optional merge rare levels)
engineered features	Numeric	created from sensors; scaled if needed

Model Choices (Why 3+ models)

Random Forest: handles non-linear patterns and interactions; relatively robust to outliers. Extra Trees: adds more randomization than RF; often performs strongly on tabular sensor datasets. XGBoost: gradient boosting model that typically performs well on structured/tabular data.

Evaluation Metric

Primary metric: Macro F1 Reason: class imbalance (minority high_activity must be evaluated fairly). Macro F1 averages F1 across classes equally.

Artifacts saved: outputs/metrics.csv outputs/best_params.json outputs/best_baseline_report.txt outputs/tuned_best_report.txt outputs/feature_importance.json models and preprocessing objects in models/

Results (Fill with your best run)

After running the pipeline:

1. Terminal output (immediate results)

• The terminal prints each model’s classification report (precision/recall/F1/support) and confusion matrix after training and evaluation.
• This allows quick comparison without opening any files.

2. Saved outputs (for submission / inspection)

• Open outputs/metrics.csv and identify the best model by macro_f1.
• Detailed reports are saved in outputs/*_report.txt (each file includes the classification report and confusion matrix).
• Best hyperparameters are saved in outputs/best_params.json.

Best model: <MODEL_NAME>
Macro F1: <VALUE>

Deployment Considerations (Real-world)

Missing values at inference: sensors can drop out → use the same imputation rules (train-fitted) and include missing flags.

Data drift: sensor calibration changes over time → monitor feature distributions and re-train when drift is detected.

Latency & compute: tree/boosting models are usually fast at inference; keep preprocessing consistent with preprocessor.joblib.

Model updates: periodic re-training using newer labeled data; compare Macro F1 across versions before promotion.