Add LowercaseTextProcessor for text normalization

[ArchishmanSengupta]

Description of the feature request:

It would be helpful to have a simple processor in genai_processors/contrib that converts all text in incoming ProcessorParts to lowercase. This would make it easier for users to build normalization pipelines.

Proposed API:
Location: genai_processors/contrib/lowercase_text_processor.py
Class: LowercaseTextProcessor
Inherits from: PartProcessor
Logic: If the part is text (is_text(part.mimetype)), convert to lowercase; else, yield unchanged. All metadata is preserved.

What problem are you trying to solve with this feature?

• Tokenization might use "Hello", "hello", and "HELLO" as different number of tokens. Lowercasing ensures that "Hello", "hello", and "HELLO" are treated the same.

• Improved search and matching

Any other information you'd like to share?

No response

[kibergus]

This sounds reasonable. I'll put this to my backlog, but it may take some time to get to it.

[ArchishmanSengupta]

@kibergus sounds good.

I am open to discussions and can help implement it myself if you assign me the same. Thanks for checking the issue out. Appreciate it.

[ArchishmanSengupta]

@kibergus any updates on this?

[kibergus]

The processor itself should be as simple as:

@processor.part_processor_function
async def lower(content: processor.ProcessorPart) -> AsyncIterable[content_api.ProcessorPart]:
  if content_api.is_text(part.mimetype):
    part.text = part.text.lower()
  yield part

But before putting it in-to the library I need to think a bit where exactly it should be and whether we should add it like that or invent something like processor.text_processor(str.lower). And there are couple of things which are likely to be more impactful e.g. adding caching support.

[ArchishmanSengupta]

@kibergus i agree with the design. I think processor.text_processor(str.lower) makes more sense, as the codebase is already using factories and higher-order functions like processor.processor_function, processor.part_processor_function to create processors from user functions.

a text_processor factory fits this pattern as this can extended to (upper, strip, etc) with a single line.

For the placing, i think we should expose it in the core via /core/text.py as lowercasing and similar transforms are fundamental for most users.

What do you think?

[ArchishmanSengupta]

@kibergus any update on this?