This PoC demonstrates a UI-semantic–aware data masking system for UI screenshots. It detects UI elements using a YOLO model, extracts text using EasyOCR + Tesseract, and masks sensitive information based on user intent.
Supported masking:
- Table Column → masks all values under a column name
- Text Field → masks only the field value
- Label Text → masks text appearing after a label
Detect UI → OCR → match user intent → mask only the required data
- YOLO detects UI elements (table column, text field, label)
- OCR extracts text inside detected regions
- User provides what to mask (column / field / label name)
- Matching text regions are masked on the image
pip install -r requirements.txtDownload the model weights from the drive. And put it on the assets/ folder. Then run the pipeline
python main.py \
--model assets/best.pt \
--image src/images/ss-1.jpeg \
--headers "Order Number" "Supplier Description" "Order Date"For the input image below, if the user wants to mask the columns Line Number, Sold To Name, Description 1, and Secondary Quality, the output will be as shown.
Input Image:
Output Image (masked):
For the input image below, if the user wants to mask the fields Order No/Type, Item Numer, Planned Effective, and Worker Order, the output will be as shown.
Input Image:
Output Image (masked):
To train the YOLO model, run the following command. Make sure to update the --data path to point to your dataset config file. To learn coco8 dataset format, check the coco8 dataset format file.
python train.py \
--data config/coco8.yaml \
--epochs 100 \
--imgsz 640 \
--batch 8- YOLO (UI element detection)
- EasyOCR + Tesseract (OCR)
- OpenCV
- Python
This is a Proof of Concept, focused on demonstrating intent-based, UI-aware masking rather than full production coverage.