Classification & text extraction with LLM and genai in MediaPipe

[fosple]

MediaPipe Solution (you are using)

genai

Programming language

Typescript

Are you willing to contribute it

Yes

Describe the feature and the current behaviour/state

Use LLMs for more than just text generation: Support Classification and text extraction use cases.

Currently only simple text generation is supported.

Will this change the current API? How?

No response

Who will benefit with this feature?

No response

Please specify the use cases for this feature

Would be awesome if you could improve the API of MediaPipe so LLMs (like Gemma 3) can not only be used for text generation, but also for more tasks. Especially when they are fine tuned for following instructions (e.g. gemma-3-270m-it).

Rough needed change to support text extraction use-case and classification use-case:

• Text extraction: Enable structured output via a supplied JSON schema.
• Classification: Add probabilities of next token. See also https://ai.google.dev/gemma/docs/agile_classifiers for more details.

This is especially useful, as one LLM can be used for multiple tasks. So edge device does not have to load many models for different tasks. Especially in Web context this can be very useful as resources are scarce.

Any Other info

No response

[HarishPermal]

Hi @fosple, thanks for the great suggestion 😉

I agree that adding structured output (JSON) and exposing token probabilities would make the LLM API much more versatile for edge and web use cases.

I have marked this as a feature request and shared it with the engineering team for consideration.