Convert the model to .litertlm

[mcuMr]

How can the model downloaded from HuggingFace be converted into .litertlm format? Are there any detailed documents available?

[sourcelite]

Hi @mcuMr, link to docs you may be looking for - https://ai.google.dev/edge/litert/conversion/pytorch/genai and https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference#pytorch_model_conversion.

btw just curious which model are you trying to convert? gemma/llama/phi? and on which hardware - android NPU, iOS metal or web browser ?

info: we see litertlm file as standardise way of running llm efficiently on-device. you can see the file litertlm as a special container that packages three essential things - tflite file is optimised model, tokeniser dictonary model uses to understand text, and lastly metadata that tells litert-lm engine which input are for text and which is output for "logits".

model downloaded from HuggingFace be converted into .litertlm format

depending upon your preference - there are three ways you can get to litertlm format -

1. download the preconverted simplest
2. generative api - it is good for custom pytorch models: litert-torch needed
3. manual bundling - existing .tflite file + tokeniser - you need tool mediapipe.tasks.python.genai.bundler

[sourcelite]

i thought it will be better to reiterate the steps for your quick ref - let me summarise steps for you to get smooth journey -

1. conversion and quantisation:

after you get your model from huggingface, use the litert torch generative api to lower the pytorch model, this will handle complex authoring required to make the transformer run faster on your mobile/devices.

import litert_torch as lrt

# Example for a Gemma model
model = lrt.generative.examples.gemma.load_hf("google/gemma-3-1b-it")
lrt.generative.convert(model, quantization="int4", output_path="model.tflite")

2. bundle creation now:
   you get the tflite file and tokenizer.model from huggingface, you will gift wrap them into litertlm format, using the mediapipe bundler.

sample -

from mediapipe.tasks.python.genai import bundler

config = bundler.BundleConfig(
    tflite_model="model.tflite",
    tokenizer_model="tokenizer.model",
    start_token="<bos>",
    stop_tokens=["<eos>", "<end_of_turn>"],
    output_filename="my_model.litertlm"
)
bundler.create_bundle(config)

I already gave the best links above, but consider looking these as well for more understanding -

convert step - https://ai.google.dev/edge/litert/conversion/pytorch/genai

optmise step - https://ai.google.dev/edge/litert/next/npu

bundle - https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference

for deploy you may like to view - text=github.com/google%2Dai%2Dedge/gallery

hope this links + info helps you. Thanks!

[snnn]

Need to use : https://github.com/google-ai-edge/LiteRT-LM/blob/main/schema/py/litertlm_builder.py

[github-actions]

Marking this issue as stale since it has been open for 7 days with no activity. This issue will be closed if no further activity occurs.

[github-actions]

This issue was closed because it has been inactive for 14 days. Please post a new issue if you need further assistance. Thanks!