Skip to content

feat: prefilter model for senstive content detection - #198

Open
Jassu7082 wants to merge 2 commits into
AnsahMohammad:mainfrom
Jassu7082:pre_filter_core
Open

feat: prefilter model for senstive content detection#198
Jassu7082 wants to merge 2 commits into
AnsahMohammad:mainfrom
Jassu7082:pre_filter_core

Conversation

@Jassu7082

Copy link
Copy Markdown

Implementation of Issue #150
Model used https://huggingface.co/urchade/gliner_small-v2.1#citation

Model Download Flow

Overview

Deep Mode relies on external assets hosted on Hugging Face.
To keep the initial APK size lightweight, these assets are downloaded dynamically at runtime.


Download Pipeline & Logic

The download process is managed by PrefilterModelDownloadService, which displays a weighted progress bar to improve user experience.

Steps

  1. Tokenizer (10%)

    • Downloads: gliner_tokenizer.json
    • Lightweight and quick to fetch
  2. Model (90%)

    • Downloads: gliner_pii_small.onnx
    • Main model file (largest portion of the download)

Flow Diagram

graph TD
    A[User Selects 'Deep' Mode] --> B{isModelDownloaded?}
    B -- "Files Exist in App Support Dir" --> C[Set Mode to 'Deep']
    B -- "Files Missing" --> D[Show _PrefilterDownloadDialog]

    D --> E[startDownload]
    E --> F[Download Tokenizer: 0% → 10%]
    F --> G[Download Model: 10% → 100%]

    G --> H{HTTP 200?}
    H -- Yes --> I[Status: Completed]
    H -- No/Error --> J[Status: Error + Log Analytics]

    I --> K[Show 'Get Started' Button]
    J --> L[Show 'Retry' Button]
Loading

Jassu7082 added 2 commits May 25, 2026 10:32
- Add PrefilterResult, PiiEntity, PiiEntityTypes models
- Add PrefilterService with regex scan (light mode) and mode persistence
- Add PrefilterMlService: GLiNER ONNX inference (deep mode)
- Add PrefilterTokenizer: BPE tokenizer for GLiNER input
- Add PrefilterModelDownloadService: HuggingFace ONNX model download
- Extend Screenshot model with prefilterStatus and prefilterReason fields
- Add onnxruntime ^1.4.0 dependency
@Jassu7082

Copy link
Copy Markdown
Author

Fail-Safe: Missing Assets

If Deep Mode is enabled but required assets are unavailable
(e.g., deleted manually or the application support directory is cleared),
the system automatically falls back to Light Mode during initialization.

if (mode == 'deep' && !isDownloaded) {
  _selectedMode = 'light';
  PrefilterService.setMode('light');
}

Source Attribution

The scanning engine uses optimized ONNX models provided by the onnx-community.


Used Dependencies

Package Version Description
onnxruntime ^1.4.0 Runs the GLiNER ONNX model on-device
http ^1.1.0 Handles streaming downloads from Hugging Face
path_provider ^2.1.0 Provides access to the Application Support Directory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant