medgemma is not working as on android. working as expected in local optimum.executorch

[kamalkraj]

@kirklandsign

git clone https://github.com/huggingface/optimum-executorch.git
cd optimum-executorch
pip install .

optimum-cli export executorch \
  --model "google/medgemma-1.5-4b-it" \
  --task "multimodal-text-to-text" \
  --recipe "xnnpack" \
  --device cpu \
  --use_custom_sdpa \
  --use_custom_kv_cache \
  --qlinear 8da4w \
  --qlinear_group_size 32 \
  --qlinear_encoder "8da4w,8da8w" \
  --qlinear_encoder_group_size 32 \
  --qembedding "8w" \
  --qembedding_encoder "8w" \
  --output_dir="medgemma-1.5-4b-it-8da4w-executorch"

local inference and verification

from optimum.executorch import ExecuTorchModelForMultiModalToText
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("google/medgemma-1.5-4b-it")
model = ExecuTorchModelForMultiModalToText.from_pretrained("medgemma-1.5-4b-it-8da4w-executorch")

messages = [{"role": "user", "content": [{"type": "text", "text": "Hi"}]}]
model.eos_token_id = 106
out = model.text_generation(
    processor, processor.tokenizer, messages, echo=False, max_seq_len=1024
)

print("Generated Text:\n", out)

output

W0127 18:23:21.123000 10266 .venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
<unused94>thought
Thinking Process:

1.  **Identify the user's input:** The user simply said "Hi".
2.  **Recognize the intent:** "Hi" is a greeting. It's a simple, common way to start a conversation or acknowledge someone's presence.
3.  **Determine the appropriate response:** A polite and friendly greeting is expected in return.
4.  **Formulate a suitable response:**
    *   Start with a reciprocal greeting: "Hi" or "Hello".
    *   Add a friendly follow-up: "How can I help you today?" or "What can I do for you?" or "How are you doing?" or "What's on your mind?"
5.  **Select the best response:** "Hi there!" is a friendly start. Adding a helpful prompt like "How can I help you today?" or "What can I do for you?" is standard for an AI assistant.
6.  **Combine and refine:** "Hi there! How can I help you today?" or "Hello! What can I do for you?" are both good options. "Hi there!" feels slightly more informal and friendly, matching the user's simple "Hi".
7.  **Final Answer:** "Hi there! How can I help you today?"<unused95>Hi there! How can I help you today?<end_of_turn>
⚠️ DISCLAIMER: Python-based perf measurements are approximate and may not match absolute speeds on Android/iOS apps. They are intended for relative comparisons—-e.g. SDPA vs. custom SDPA, FP16 vs. FP32—-so you can gauge performance improvements from each optimization step. For end-to-end, platform-accurate benchmarks, please use the official ExecuTorch apps:
  • iOS:     https://github.com/pytorch/executorch/tree/main/extension/benchmark/apple/Benchmark
  • Android: https://github.com/pytorch/executorch/tree/main/extension/benchmark/android/benchmark

PyTorchObserver {"prompt_tokens": 10, "generated_tokens": 288, "model_load_start_ms": 0, "model_load_end_ms": 0, "inference_start_ms": 1769518409191, "token_encode_end_ms": 1769518409197, "model_execution_start_ms": 0, "model_execution_end_ms": 0, "inference_end_ms": 1769518433113, "prompt_eval_end_ms": 1769518411351, "first_token_ms": 1769518411429, "aggregate_sampling_time_ms": 23827, "SCALING_FACTOR_UNITS_PER_SECOND": 1000}
        Prompt Tokens: 10 Generated Tokens: 288
        Model Load Time:                0.000000 (seconds)
        Total inference time:           23.922000 (seconds)              Rate:  12.039127 (tokens/second)
                Prompt evaluation:      2.160000 (seconds)               Rate:  4.629630 (tokens/second)
                Generated 288 tokens:   21.762000 (seconds)              Rate:  13.234078 (tokens/second)
        Time to first generated token:  2.238000 (seconds)
        Sampling time over 298 tokens:  23.827000 (seconds)
Generated Text:
 thoughtThinking Process:1.**Identify the user's input:** The user simply said "Hi".2.**Recognize the intent:** "Hi" is a greeting. It's a simple, common way to start a conversation or acknowledge someone's presence.3.**Determine the appropriate response:** A polite and friendly greeting is expected in return.4.**Formulate a suitable response:***Start with a reciprocal greeting: "Hi" or "Hello".*Add a friendly follow-up: "How can I help you today?" or "What can I do for you?" or "How are you doing?" or "What's on your mind?"5.**Select the best response:** "Hi there!" is a friendly start. Adding a helpful prompt like "How can I help you today?" or "What can I do for you?" is standard for an AI assistant.6.**Combine and refine:** "Hi there! How can I help you today?" or "Hello! What can I do for you?" are both good options. "Hi there!" feels slightly more informal and friendly, matching the user's simple "Hi".7.**Final Answer:** "Hi there! How can I help you today?"Hi there! How can I help you today?

Copy the model and tokenizer

hf download google/medgemma-1.5-4b-it --local-dir medgemma-1.5-4b-it
cp medgemma-1.5-4b-it/tokenizer.model medgemma-1.5-4b-it-8da4w-executorch/

adb shell mkdir -p /data/local/tmp/llama/
adb push medgemma-1.5-4b-it-8da4w-executorch/tokenizer.model /data/local/tmp/llama/
adb push medgemma-1.5-4b-it-8da4w-executorch/model.pte /data/local/tmp/llama/ 

One device generartion is not working as expected. Following the same step for gemma3-4b-it working as expected.

Screen_recording_20260127_182610.mp4

[kirklandsign]

For local inference have you tried executorch multimodal runner?

[kamalkraj]

@kirklandsign

The python MultimodalRunner is throwing an error. This is how it is run

git clone https://github.com/pytorch/executorch.git

docker run -it \
  -v /Users/kamalkraj/Documents/Github/executorch:/executorch \
  --platform linux/amd64 \
  python:3.12-trixie bash

cd /executorch && ./install_executorch.sh 

from executorch.extension.llm.runner import (
    GenerationConfig, MultimodalRunner, 
    make_text_input, make_image_input, make_audio_input
)
import torch

# Create a multimodal runner
runner = MultimodalRunner(
    model_path="medgemma-1.5-4b-it-8da4w-executorch/model.pte",
    tokenizer_path="medgemma-1.5-4b-it-8da4w-executorch/tokenizer.model"
)

# Create multimodal inputs
inputs = []
inputs.append(make_text_input("What do you see in this image?"))

# Add image from torch tensor (supports both CHW and HWC formats)
image_tensor = torch.randint(0, 255, (3, 224, 224), dtype=torch.uint8)  # CHW format
inputs.append(make_image_input(image_tensor))

# Configure generation
config = GenerationConfig(
    max_new_tokens=100,
    temperature=0.7,
    echo=False
)

# Generate with streaming output
def token_callback(token: str):
    print(token, end='', flush=True)

def stats_callback(stats):
    print(f"\n[Stats] Generated {stats.num_generated_tokens} tokens")
    inference_time = stats.inference_end_ms - stats.inference_start_ms
    if inference_time > 0:
        tokens_per_sec = stats.num_generated_tokens * 1000 / inference_time
        print(f"[Stats] Speed: {tokens_per_sec:.1f} tokens/sec")

runner.generate(inputs, config, token_callback, stats_callback)

output

I tokenizers:regex.cpp:27] Registering override fallback regex
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:55] invalid tiktoken line: 
[llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
[llm_runner_helper.cpp:293] Reading metadata from model
[llm_runner_helper.cpp:133] Metadata: use_kv_cache = 1
[llm_runner_helper.cpp:127] Method get_max_context_len not found, using the default value 128
[llm_runner_helper.cpp:133] Metadata: get_max_context_len = 128
[llm_runner_helper.cpp:133] Metadata: get_max_seq_len = 2048
[llm_runner_helper.cpp:133] Metadata: use_sdpa_with_kv_cache = 1
[llm_runner_helper.cpp:133] Metadata: enable_dynamic_shape = 1
[llm_runner_helper.cpp:141] Setting kMaxContextLen to kMaxSeqLen value: 2048
[cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu0/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu1/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu2/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu3/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu4/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu5/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu6/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu7/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu8/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu9/regs/identification/midr_el1
[operator_registry.cpp:256] kernel 'quantized_decomposed::embedding_byte.out' not found.
[operator_registry.cpp:257] dtype: 1 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 6 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 4 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 6 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] 2,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 6 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] 2,
[operator_registry.cpp:257] ]
[method.cpp:784] Missing operator: [0] quantized_decomposed::embedding_byte.out
[method.cpp:1048] There are 1 instructions don't have corresponding operator registered. See logs for details
Traceback (most recent call last):
  File "/executorch/model_infer.py", line 39, in <module>
    runner.generate(inputs, config, token_callback, stats_callback)
RuntimeError: Generation failed

[larryliu0820]

@kirklandsign

The python MultimodalRunner is throwing an error. This is how it is run

git clone https://github.com/pytorch/executorch.git

docker run -it
-v /Users/kamalkraj/Documents/Github/executorch:/executorch
--platform linux/amd64
python:3.12-trixie bash

cd /executorch && ./install_executorch.sh
from executorch.extension.llm.runner import (
GenerationConfig, MultimodalRunner,
make_text_input, make_image_input, make_audio_input
)
import torch

Create a multimodal runner

runner = MultimodalRunner(
model_path="medgemma-1.5-4b-it-8da4w-executorch/model.pte",
tokenizer_path="medgemma-1.5-4b-it-8da4w-executorch/tokenizer.model"
)

Create multimodal inputs

inputs = []
inputs.append(make_text_input("What do you see in this image?"))

Add image from torch tensor (supports both CHW and HWC formats)

image_tensor = torch.randint(0, 255, (3, 224, 224), dtype=torch.uint8) # CHW format
inputs.append(make_image_input(image_tensor))

Configure generation

config = GenerationConfig(
max_new_tokens=100,
temperature=0.7,
echo=False
)

Generate with streaming output

def token_callback(token: str):
print(token, end='', flush=True)

def stats_callback(stats):
print(f"\n[Stats] Generated {stats.num_generated_tokens} tokens")
inference_time = stats.inference_end_ms - stats.inference_start_ms
if inference_time > 0:
tokens_per_sec = stats.num_generated_tokens * 1000 / inference_time
print(f"[Stats] Speed: {tokens_per_sec:.1f} tokens/sec")

runner.generate(inputs, config, token_callback, stats_callback)

output

I tokenizers:regex.cpp:27] Registering override fallback regex
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:55] invalid tiktoken line:
[llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
[llm_runner_helper.cpp:293] Reading metadata from model
[llm_runner_helper.cpp:133] Metadata: use_kv_cache = 1
[llm_runner_helper.cpp:127] Method get_max_context_len not found, using the default value 128
[llm_runner_helper.cpp:133] Metadata: get_max_context_len = 128
[llm_runner_helper.cpp:133] Metadata: get_max_seq_len = 2048
[llm_runner_helper.cpp:133] Metadata: use_sdpa_with_kv_cache = 1
[llm_runner_helper.cpp:133] Metadata: enable_dynamic_shape = 1
[llm_runner_helper.cpp:141] Setting kMaxContextLen to kMaxSeqLen value: 2048
[cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu0/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu1/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu2/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu3/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu4/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu5/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu6/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu7/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu8/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu9/regs/identification/midr_el1
[operator_registry.cpp:256] kernel 'quantized_decomposed::embedding_byte.out' not found.
[operator_registry.cpp:257] dtype: 1 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 6 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 4 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 6 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] 2,
[operator_registry.cpp:257] ]
[operator_registry.cpp:257] dtype: 6 | dim order: [
[operator_registry.cpp:257] 0,
[operator_registry.cpp:257] 1,
[operator_registry.cpp:257] 2,
[operator_registry.cpp:257] ]
[method.cpp:784] Missing operator: [0] quantized_decomposed::embedding_byte.out
[method.cpp:1048] There are 1 instructions don't have corresponding operator registered. See logs for details
Traceback (most recent call last):
File "/executorch/model_infer.py", line 39, in
runner.generate(inputs, config, token_callback, stats_callback)
RuntimeError: Generation failed

Can you try import quantized?

from executorch.kernels import quantized

[kirklandsign]

I got this error

2026-01-27 20:12:10.135 23257-25161 ExecuTorch              com.example.executorchllamademo      E  Unhandled node type: 
2026-01-27 20:12:10.135 23257-25161 ExecuTorch              com.example.executorchllamademo      E  XNNCompiler::compileModel failed: 0x11
2026-01-27 20:12:10.135 23257-25161 ExecuTorch              com.example.executorchllamademo      E  Init failed for backend XnnpackBackend: 0x11
2026-01-27 20:12:10.135 23257-25161 ExecuTorch              com.example.executorchllamademo      E  Error, attempted to delete packed data from the cache but the cache is not finalized
2026-01-27 20:12:10.135 23257-25161 ExecuTorch              com.example.executorchllamademo      E  Error, attempted to delete packed data from the cache but the cache is not finalized
2026-01-27 20:12:10.135 23257-25161 ExecuTorch              com.example.executorchllamademo      E  Error, attempted to delete packed data from the cache but the cache is not finalized

[kamalkraj]

@larryliu0820 @kirklandsign
running after the additional imports. output is still wrong

from executorch.extension.llm.custom_ops import custom_ops
from executorch.kernels import quantized
from executorch.extension.llm.runner import (
    GenerationConfig, MultimodalRunner, 
    make_text_input, make_image_input, make_audio_input
)
import torch

# Create a multimodal runner
runner = MultimodalRunner(
    model_path="medgemma-1.5-4b-it-8da4w-executorch/model.pte",
    tokenizer_path="medgemma-1.5-4b-it-8da4w-executorch/tokenizer.model"
)

# Create multimodal inputs
inputs = []
inputs.append(make_text_input("<bos><start_of_turn>user\nHi?<end_of_turn>\n<start_of_turn>model\n"))

# # Add image from torch tensor (supports both CHW and HWC formats)
# image_tensor = torch.randint(0, 255, (3, 896, 896), dtype=torch.float32)  # CHW format
# inputs.append(make_image_input(image_tensor))

# Configure generation
config = GenerationConfig(
    max_new_tokens=100,
    temperature=0.0,
    echo=False,
    num_bos=1,
)

# Generate with streaming output
def token_callback(token: str):
    print(token, end='', flush=True)

def stats_callback(stats):
    print(f"\n[Stats] Generated {stats.num_generated_tokens} tokens")
    inference_time = stats.inference_end_ms - stats.inference_start_ms
    if inference_time > 0:
        tokens_per_sec = stats.num_generated_tokens * 1000 / inference_time
        print(f"[Stats] Speed: {tokens_per_sec:.1f} tokens/sec")

runner.generate(inputs, config, token_callback, stats_callback)

W0128 10:35:45.410000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.410000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.410000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.410000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.411000 33819 .venv/lib/python3.12/site-packages/torch/utils/flop_counter.py:45] triton not found; flop counting will not work for triton kernels
W0128 10:35:45.707000 33819 .venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
I tokenizers:regex.cpp:27] Registering override fallback regex
E tokenizers:hf_tokenizer.cpp:82] Error parsing json file: [json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - invalid literal; last read: '<U+000A><U+000E>'
E tokenizers:tiktoken.cpp:59] invalid tiktoken line: 
[llm_runner_helper.cpp:77] Loaded Sentencepiece tokenizer
[llm_runner_helper.cpp:293] Reading metadata from model
[llm_runner_helper.cpp:133] Metadata: use_sdpa_with_kv_cache = 1
[llm_runner_helper.cpp:133] Metadata: use_kv_cache = 1
[llm_runner_helper.cpp:131] Method get_max_context_len not found, using the default value 128
[llm_runner_helper.cpp:133] Metadata: get_max_context_len = 128
[llm_runner_helper.cpp:133] Metadata: get_max_seq_len = 2048
[llm_runner_helper.cpp:133] Metadata: enable_dynamic_shape = 1
[llm_runner_helper.cpp:144] Setting kMaxContextLen to kMaxSeqLen value: 2048
[cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
[multimodal_runner.cpp:122] RSS after loading model: 0.000000 MiB (0 if unsupported)
[multimodal_runner.cpp:148] Prefilling input 0/1, type: text
<unused94><unused94>[multimodal_runner.cpp:177] RSS after multimodal input processing: 0.000000 MiB (0 if unsupported)
[multimodal_runner.cpp:189] Max new tokens resolved: 100, pos_ 13, max_context_len 2048
modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>modelmodel

<unused94><unused94>
PyTorchObserver {"prompt_tokens":13,"generated_tokens":99,"model_load_start_ms":1769576747014,"model_load_end_ms":1769576749676,"inference_start_ms":1769576749676,"inference_end_ms":1769576759480,"prompt_eval_end_ms":1769576749968,"first_token_ms":1769576749968,"aggregate_sampling_time_ms":44,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
[stats.h:143]   Prompt Tokens: 13    Generated Tokens: 99
[stats.h:149]   Model Load Time:                2.662000 (seconds)
[stats.h:159]   Total inference time:           9.804000 (seconds)               Rate:  10.097919 (tokens/second)
[stats.h:167]           Prompt evaluation:      0.292000 (seconds)               Rate:  44.520548 (tokens/second)
[stats.h:178]           Generated 99 tokens:    9.512000 (seconds)               Rate:  10.407906 (tokens/second)
[stats.h:186]   Time to first generated token:  0.292000 (seconds)
[stats.h:193]   Sampling time over 112 tokens:  0.044000 (seconds)

[Stats] Generated 99 tokens
[Stats] Speed: 10.1 tokens/sec

Running the model directly using transformers

# Make sure to install the accelerate library first via `pip install accelerate`
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "google/medgemma-1.5-4b-it"

model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Hi"}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=1024, do_sample=False,eos_token_id=[1,106])
    generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=False)
print(decoded)

Output

Loading weights: 100%|██████| 883/883 [00:00<00:00, 5628.18it/s, Materializing param=model.vision_tower.vision_model.post_layernorm.weight]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Setting `pad_token_id` to `eos_token_id`:1 for open-end generation.
<unused94>thought
My thinking process to respond to "Hi" goes like this:

1.  **Identify the input:** The user said "Hi". This is a standard, informal greeting.

2.  **Determine the user's intent:** The user is initiating a conversation, likely expecting a response and possibly indicating they want to ask something or just chat.

3.  **Formulate a suitable response:** A simple, polite, and welcoming response is appropriate.
    *   **Acknowledge the greeting:** The most direct response is to return the greeting. "Hi" or "Hello" are standard.
    *   **Offer assistance/engagement:** Since I'm an AI assistant, it's helpful to signal my readiness to help or engage further. Phrases like "How can I help you today?", "What can I do for you?", or "How are you doing?" work well.
    *   **Keep it concise and friendly:** The tone should match the user's informal greeting.

4.  **Select the best options:**
    *   "Hi there!" - Friendly and returns the greeting.
    *   "Hello!" - Standard and polite.
    *   "Hi! How can I help you today?" - Returns the greeting and immediately offers assistance, which is my primary function.
    *   "Hello! What can I do for you?" - Similar to the above, slightly different phrasing.
    *   "Hi! How are you?" - Friendly, but less direct about my purpose as an assistant. Might be better if the user seems to want to chat more first.

5.  **Choose the most versatile and helpful response:** "Hi there! How can I help you today?" is a good balance. It's friendly, returns the greeting, and clearly states my purpose. "Hello! How can I help you today?" is also excellent.

6.  **Final Output Construction:** Combine the chosen greeting and the offer of assistance.

    *   *Initial thought:* "Hi! How can I help?" (A bit too short)
    *   *Second thought:* "Hello! What can I do for you?" (Good, but maybe slightly formal)
    *   *Refined thought:* "Hi there! How can I help you today?" (Friendly, clear, standard)
    *   *Alternative:* "Hello! How may I assist you?" (Slightly more formal)
    *   *Another alternative:* "Hi! What's on your mind?" (More conversational)

7.  **Select the final response:** "Hi there! How can I help you today?" seems like a solid, standard, and helpful response for this initial greeting.<unused95>Hi there! How can I help you today?<end_of_turn>

Running model using ExecuTorchModule pybindings

"""
Standalone script to run ExecuTorch model inference using just ExecuTorchModule and torch.
Uses SentencePiece tokenizer for encoding/decoding.

Model weights: medgemma-1.5-4b-it-8da4w-executorch/
"""

import torch
import sentencepiece as spm
from typing import List, Optional

from executorch.extension.pybindings.portable_lib import (
    ExecuTorchModule,
    _load_for_executorch,
)
from executorch.extension.llm.custom_ops import custom_ops  # noqa - required for custom ops
from executorch.kernels import quantized  # noqa - required for quantized models
from torch.ao.quantization.fx._decomposed import quantized_decomposed_lib  # noqa


def load_model(pte_path: str) -> ExecuTorchModule:
    """Load an ExecuTorch model from a .pte file."""
    model = _load_for_executorch(pte_path)
    return model


def load_tokenizer(tokenizer_path: str) -> spm.SentencePieceProcessor:
    """Load a SentencePiece tokenizer."""
    sp = spm.SentencePieceProcessor()
    sp.Load(tokenizer_path)
    return sp


def get_model_metadata(model: ExecuTorchModule) -> dict:
    """Extract metadata from the loaded model."""
    metadata = {}
    method_names = model.method_names()

    # Print available methods for debugging
    print(f"Available model methods: {method_names}")

    # Get decoder-related metadata
    if "get_max_seq_len" in method_names:
        metadata["max_cache_size"] = model.run_method("get_max_seq_len")[0]
    if "get_bos_id" in method_names:
        metadata["bos_token_id"] = model.run_method("get_bos_id")[0]
    if "get_eos_id" in method_names:
        metadata["eos_token_id"] = model.run_method("get_eos_id")[0]

    # Try to get multiple EOS IDs
    eos_ids = []
    if "get_eos_ids" in method_names:
        eos_ids = list(model.run_method("get_eos_ids"))
    elif "get_eos_id" in method_names:
        eos_ids = [model.run_method("get_eos_id")[0]]
    metadata["eos_token_ids"] = eos_ids

    # Check for dynamic shapes
    if "enable_dynamic_shape" in method_names:
        metadata["enable_dynamic_shape"] = model.run_method("enable_dynamic_shape")[0]

    return metadata


def forward(
    model: ExecuTorchModule,
    input_ids: torch.Tensor,
    cache_position: torch.Tensor,
) -> torch.Tensor:
    """
    Forward pass of the model using token_embedding + text_decoder.

    Args:
        model: The loaded ExecuTorchModule
        input_ids: Token IDs tensor [batch_size, seq_len]
        cache_position: Position tensor for KV cache

    Returns:
        Logits tensor from the model
    """
    # Get token embeddings
    token_embeddings = model.run_method("token_embedding", (input_ids,))[0]

    # Run text decoder
    logits = model.run_method("text_decoder", (token_embeddings, cache_position))[0]

    return logits


def generate(
    model: ExecuTorchModule,
    prompt_tokens: List[int],
    eos_token_ids: List[int],
    max_cache_size: int,
    max_seq_len: Optional[int] = None,
    tokenizer: Optional[spm.SentencePieceProcessor] = None,
) -> List[int]:
    """
    Generate tokens from a prompt using the ExecuTorch model.

    Args:
        model: The loaded ExecuTorchModule
        prompt_tokens: List of input token IDs
        eos_token_ids: List of end-of-sequence token IDs
        max_cache_size: Maximum sequence length supported by the cache
        max_seq_len: Maximum sequence length for generation
        tokenizer: Optional SentencePiece tokenizer for streaming decode

    Returns:
        List of generated token IDs
    """
    device = torch.device("cpu")

    if max_seq_len is None:
        max_seq_len = max_cache_size
    elif max_seq_len > max_cache_size:
        print(f"Warning: max_seq_len={max_seq_len} > max_cache_size={max_cache_size}. Truncating.")
        max_seq_len = max_cache_size

    generated_tokens = []

    method_names = model.method_names()
    has_dynamic_shape = "enable_dynamic_shape" in method_names and model.run_method("enable_dynamic_shape")[0]

    if has_dynamic_shape:
        # Use parallel prefill - process entire prompt at once
        input_ids = torch.tensor(prompt_tokens, dtype=torch.long, device=device).unsqueeze(0)
        cache_position = torch.arange(len(prompt_tokens), dtype=torch.long, device=device)
        logits = forward(model, input_ids, cache_position)
        next_token = torch.argmax(logits, dim=-1)[0, -1].item()
    else:
        # Sequential prefill - process one token at a time
        for i, token in enumerate(prompt_tokens):
            input_ids = torch.tensor([[token]], dtype=torch.long, device=device)
            cache_position = torch.tensor([i], dtype=torch.long, device=device)
            logits = forward(model, input_ids, cache_position)
        next_token = torch.argmax(logits[:, -1, :], dim=-1).item()

    generated_tokens = prompt_tokens + [next_token]

    if tokenizer is not None:
        print(tokenizer.Decode([next_token]), end="", flush=True)

    # Token-by-token generation
    while len(generated_tokens) < max_seq_len:
        input_ids = torch.tensor([[next_token]], dtype=torch.long, device=device)
        cache_position = torch.tensor([len(generated_tokens) - 1], dtype=torch.long, device=device)

        logits = forward(model, input_ids, cache_position)
        next_token = torch.argmax(logits[:, -1, :], dim=-1).item()
        generated_tokens.append(next_token)

        if tokenizer is not None:
            print(tokenizer.Decode([next_token]), end="", flush=True)

        if next_token in eos_token_ids:
            break

    if tokenizer is not None:
        print()  # Newline after generation

    return generated_tokens


def run_inference(
    pte_path: str,
    tokenizer_path: str,
    prompt: str,
    max_new_tokens: int = 256,
    stream: bool = True,
) -> str:
    """
    High-level function to run text inference.

    Args:
        pte_path: Path to the .pte model file
        tokenizer_path: Path to the SentencePiece .model file
        prompt: Input prompt string
        max_new_tokens: Maximum number of new tokens to generate
        stream: Whether to stream output tokens

    Returns:
        Generated text string
    """
    # Load model and tokenizer
    model = load_model(pte_path)
    tokenizer = load_tokenizer(tokenizer_path)

    # Get metadata
    metadata = get_model_metadata(model)
    print(f"Model metadata: {metadata}")

    # Tokenize prompt with BOS token
    prompt_tokens = tokenizer.Encode(prompt, add_bos=True)
    print(f"Prompt tokens ({len(prompt_tokens)}): {prompt_tokens}")

    # Calculate max sequence length
    max_seq_len = len(prompt_tokens) + max_new_tokens

    # Get EOS token IDs
    eos_token_ids = metadata.get("eos_token_ids", [metadata.get("eos_token_id", 1)])
    if not isinstance(eos_token_ids, list):
        eos_token_ids = [eos_token_ids]

    # For Gemma models, add <end_of_turn> token (107) as EOS if not already present
    # Token 107 corresponds to <end_of_turn> in Gemma tokenizer
    gemma_end_of_turn = 106
    if gemma_end_of_turn not in eos_token_ids:
        eos_token_ids.append(gemma_end_of_turn)

    print(f"EOS token IDs: {eos_token_ids}")

    # Generate
    generated_tokens = generate(
        model=model,
        prompt_tokens=prompt_tokens,
        eos_token_ids=eos_token_ids,
        max_cache_size=metadata.get("max_cache_size", 2048),
        max_seq_len=max_seq_len,
        tokenizer=tokenizer if stream else None,
    )

    # Decode output (excluding prompt)
    output_tokens = generated_tokens[len(prompt_tokens) :]
    output_text = tokenizer.Decode(output_tokens)

    return output_text


# Example usage
if __name__ == "__main__":
    # Model paths
    MODEL_DIR = "medgemma-1.5-4b-it-8da4w-executorch"
    PTE_PATH = f"{MODEL_DIR}/model.pte"
    TOKENIZER_PATH = f"{MODEL_DIR}/tokenizer.model"

    # Input prompt in Gemma chat format
    PROMPT = "<bos><start_of_turn>user\nHi?<end_of_turn>\n<start_of_turn>model\n"

    print(f"Loading model from: {PTE_PATH}")
    print(f"Loading tokenizer from: {TOKENIZER_PATH}")
    print(f"Prompt: {repr(PROMPT)}")
    print("-" * 50)

    output = run_inference(
        pte_path=PTE_PATH,
        tokenizer_path=TOKENIZER_PATH,
        prompt=PROMPT,
        max_new_tokens=256,
        stream=True,
    )

    print("-" * 50)
    print(f"Generated output: {output}")

Output

W0128 10:41:07.213000 35770 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
Loading model from: medgemma-1.5-4b-it-8da4w-executorch/model.pte
Loading tokenizer from: medgemma-1.5-4b-it-8da4w-executorch/tokenizer.model
Prompt: '<bos><start_of_turn>user\nHi?<end_of_turn>\n<start_of_turn>model\n'
--------------------------------------------------
Available model methods: ['size', 'rescale_factor', 'modality', 'processor_class', 'image_seq_length', 'image_processor_type', 'do_rescale', 'do_convert_rgb', 'enable_dynamic_shape', 'use_sdpa_with_kv_cache', 'vision_token_id', 'sliding_window', 'get_max_seq_len', 'get_max_batch_size', 'get_vocab_size', 'image_mean', 'use_kv_cache', 'get_n_layers', 'get_n_kv_heads', 'resample', 'do_resize', 'get_head_dim', 'image_std', 'get_eos_id', 'do_normalize', 'get_bos_id', 'get_dtype', 'vision_encoder', 'token_embedding', 'text_decoder']
Model metadata: {'max_cache_size': 2048, 'bos_token_id': 2, 'eos_token_id': 1, 'eos_token_ids': [1], 'enable_dynamic_shape': True}
Prompt tokens (14): [2, 236820, 46757, 236813, 105, 2364, 107, 10979, 236881, 106, 107, 105, 4368, 107]
EOS token IDs: [1, 106]
<unused94>thought
Thinking Process:

1.  **Identify the input:** The user said "Hi?".
2.  **Recognize the intent:** This is a greeting. It's a simple, informal way to start a conversation.
3.  **Determine the appropriate response:** A polite and friendly greeting is expected.
4.  **Formulate a response:**
    *   Acknowledge the greeting: "Hi!" or "Hello!"
    *   Offer assistance: Since I'm an AI assistant, it's helpful to ask how I can help. Phrases like "How can I help you today?" or "What can I do for you?" are suitable.
5.  **Combine the elements:** "Hi there! How can I help you today?" or "Hello! What can I do for you?"
6.  **Select the best option:** "Hi there! How can I help you today?" sounds friendly and direct.<unused95>Hi there! How can I help you today?<end_of_turn>
--------------------------------------------------
Generated output: <unused94>thought
Thinking Process:

1.  **Identify the input:** The user said "Hi?".
2.  **Recognize the intent:** This is a greeting. It's a simple, informal way to start a conversation.
3.  **Determine the appropriate response:** A polite and friendly greeting is expected.
4.  **Formulate a response:**
    *   Acknowledge the greeting: "Hi!" or "Hello!"
    *   Offer assistance: Since I'm an AI assistant, it's helpful to ask how I can help. Phrases like "How can I help you today?" or "What can I do for you?" are suitable.
5.  **Combine the elements:** "Hi there! How can I help you today?" or "Hello! What can I do for you?"
6.  **Select the best option:** "Hi there! How can I help you today?" sounds friendly and direct.<unused95>Hi there! How can I help you today?<end_of_turn>