Extract Alt Text and Handwritten Text from Images Using ML Model

[rithulkamesh]

We're building a smarter, memory-efficient document processing pipeline with the following goals:

• Extract alt text and handwritten content from images in documents using lightweight ML models.
• Use lighter vision-language models like Janus-1.3B, Mistral-small, or Gemma 3B (quantized where possible).
• Integrate QLoRA fine-tuning if a suitable dataset is available.
• Explore reasoning-based summarization pipelines:
  • Generate a full summary from PDFs or scanned docs using AI (0-loss goal).
  • Pass entire summaries directly into lightweight models for context handling.

Pipeline Goals:

• Alt text extraction using image captioning models.
• Handwriting extraction via OCR/HWR (e.g., TrOCR or PaddleOCR).

[rithulkamesh]

I'm proposing moving from pytesseract to Granite 3.2 Vision, consolidating

class RegionType(Enum):
    """Enumeration of supported document region types.

    Defines the different types of regions that can be detected within a document:
    - TEXT: Regions containing textual content
    - EQUATION: Regions containing mathematical equations
    - IMAGE: Regions containing images or graphics
    - HANDWRITING: Regions containing handwritten content
    """

    TEXT = auto()
    EQUATION = auto()
    IMAGE = auto()
    HANDWRITING = auto()
    UNCLASSIFIED = auto()

    def to_json(self):
        return self.name

into

class RegionType(Enum):
    """Enumeration of supported document region types.

    Defines the different types of regions that can be detected within a document:
    - TEXT: Regions containing textual content
    - EQUATION: Regions containing mathematical equations
    - VISUAL_CONTENT: Regions containing viewable content but not textual content, can be graphs, tables, handwriting etc.
    """

    TEXT = auto()
    EQUATION = auto()
    VISUAL_CONTENT = auto()
    UNCLASSIFIED = auto()

    def to_json(self):
        return self.name