[RISC-V] Benchmarking IREE vs ExecuTorch / ONNX Runtime / LiteRT

[dimp-pl]

This issue stores the results of benchmarking IREE vs ExecuTorch/LiteRT/ONNX Runtime on RISC-V CPUs.
Happy to rerun or adjust the benchmark to run more models.

What was measured

Board spec: SpacemiT K3, 8x X100 @ 2.2 GHz (VLEN-256) + 8x A100 @ 1.8 GHz (VLEN-1024), kernel 6.18.3-1.0.5.4.

Additional notes:

• performance governor was pinned on both X100 and A100 clusters, CPU temp 57-66C measured (no throttling).
• Every result produced by IREE was verified against an ONNX Runtime reference to be correct.

Data types i wanted to measure inference on:

• fp32
• fp16
• int8
• bf16

Models

I've tried to choose a diverse set of models to cover both convolution-heavy models, as well as larger models like transformers.

model	source (pretrained weights)	regime
MobileNetV2	torchvision MobileNet_V2_Weights.IMAGENET1K_V2	depthwise CNN
MobileNetV3-Small	torchvision MobileNet_V3_Small_Weights.IMAGENET1K_V1	depthwise CNN
EfficientNet-B0	torchvision EfficientNet_B0_Weights.IMAGENET1K_V1	depthwise + SE
ShuffleNetV2-x1.0	torchvision ShuffleNet_V2_X1_0_Weights.IMAGENET1K_V1	depthwise CNN
ResNet18	torchvision ResNet18_Weights.IMAGENET1K_V1	dense conv
ResNet50	torchvision ResNet50_Weights.IMAGENET1K_V2	dense conv
VGG16	torchvision VGG16_Weights.IMAGENET1K_V1	dense conv
ViT-B/16	torchvision ViT_B_16_Weights.IMAGENET1K_V1	transformer
DeiT-Tiny	timm deit_tiny_patch16_224.fb_in1k	transformer
MLP-Mixer-B/16	timm mixer_b16_224.goog_in21k_ft_in1k	token/channel MLP
DeepLabV3-MobileNetV3-Large @512²	torchvision DeepLabV3_MobileNet_V3_Large_Weights.COCO_WITH_VOC_LABELS_V1	dense prediction
GEMV tower	synthetic — 12x Linear(2048,2048) + GELU, seeded	batch-1 LLM-decode proxy

Preparation pipeline: torch.onnx.export(opset_version=17) batch-1 static ->
iree.compiler.tools.import_onnx -> iree-compile. --externalize-params was used
for the four large models. int8 = ORT quantize_static, QDQ, per-channel, QInt8 weights / QUInt8
activations, 16 seeded calibration images — the same quantized .onnx feeds IREE and
ORT, while ExecuTorch/LiteRT use PT2E on the same data since they cannot ingest QDQ.

IREE configuration

I used the blog post from last week to guide the compiler configuration i used.

Common target flags:

--iree-hal-target-backends=llvm-cpu
--iree-llvmcpu-target-triple=riscv64-unknown-linux-gnu
--iree-llvmcpu-target-cpu=generic-rv64
--iree-llvmcpu-target-abi=lp64d
--iree-llvmcpu-target-cpu-features=+m,+a,+f,+d,+c,+v,+zvl256b   # +zfh,+zvfh for fp16

The cross-stack tables below use the best measured combination:

--iree-opt-data-tiling=true
--iree-llvmcpu-enable-ukernels=none
--iree-global-opt-use-im2col-for-convs=true

Note on using --iree-llvmcpu-stack-allocation-limit=4194304

Some of the models failed compilation with 'func.func' op exceeded stack allocation limit error.
I worked around it by setting --iree-llvmcpu-stack-allocation-limit=4194304 for those models.

Models affected:

• DeiT-Tiny fp32/fp16
• ResNet18 int8
• ResNet50 int8

Results

Every datatype has both an 8 threads and a 1 thread table; conclusions quote the
8-thread figures unless stated. Bold = fastest stack in the row. — = did not run,
noted below in the "what would not run" section.

fp32

8 threads (ms/inference, lower is better)

model	IREE	ExecuTorch	ONNX Runtime	LiteRT
mobilenet_v3_small	8.23	4.36	6.26	4.11
shufflenet_v2_x1_0	8.61	10.6	10.8	6.56
mobilenet_v2	34.6	9.62	20.7	9.90
deit_tiny	—	124.3	52.0	32.5
efficientnet_b0	32.3	20.8	25.5	25.6
resnet18	48.6	34.5	67.5	35.1
resnet50	95.3	86.0	143.9	82.8
mixer_b16	263.0	351.4	296.6	296.7
vit_b_16	440.0	826.9	407.7	405.0
vgg16	518.0	241.2	661.9	285.5
deeplab_v3	821.0	220.8	361.0	233.1
gemv_tower	10.7	12.5	10.7	11.0

1 thread (ms/inference, lower is better)

model	IREE	ExecuTorch	ONNX Runtime	LiteRT
mobilenet_v3_small	29.2	20.7	19.9	18.7
shufflenet_v2_x1_0	32.6	55.2	46.6	23.7
mobilenet_v2	107.0	48.3	133.5	48.9
deit_tiny	—	165.6	214.6	103.3
efficientnet_b0	172.0	120.7	136.9	107.7
resnet18	208.0	131.1	417.6	131.0
resnet50	397.0	300.1	936.2	300.2
mixer_b16	1274	1586	1868	1250
vit_b_16	2226	2161	2605	1769
vgg16	2072	1241	3715	1240
deeplab_v3	3114	1140	2069	1084
gemv_tower	38.2	28.5	24.2	33.5

DeiT-Tiny does not build in this configuration, with data tiling off it is 86.4 ms (8T).

fp16

8 threads (ms/inference, lower is better)

model	IREE	ExecuTorch	ONNX Runtime	LiteRT
mobilenet_v3_small	4.68	5.95	6.35	—
shufflenet_v2_x1_0	5.05	27.6	12.0	—
mobilenet_v2	28.5	5.39	21.5	—
deit_tiny	—	148.4	65.6	—
efficientnet_b0	23.7	120.8	25.6	—
resnet18	27.3	86.4	67.0	—
resnet50	48.8	107.3	142.7	—
mixer_b16	140.0	701.9	309.8	—
vit_b_16	189.0	722.3	431.3	—
vgg16	237.0	381.1	663.6	—
deeplab_v3	435.0	420.5	364.4	—
gemv_tower	5.31	7.28	11.1	—

1 thread (ms/inference, lower is better)

model	IREE	ExecuTorch	ONNX Runtime	LiteRT
mobilenet_v3_small	23.2	20.7	20.1	—
shufflenet_v2_x1_0	23.0	37.9	48.2	—
mobilenet_v2	84.6	27.6	99.3	—
deit_tiny	—	217.3	243.6	—
efficientnet_b0	141.0	193.2	137.5	—
resnet18	117.0	138.0	418.0	—
resnet50	219.0	214.3	935.9	—
mixer_b16	630.0	1129	1977	—
vit_b_16	946.0	1501	2731	—
vgg16	931.0	860.4	3713	—
deeplab_v3	2081	960.4	2082	—
gemv_tower	13.6	21.4	24.5	—

int8

8 threads (ms/inference, lower is better)

model	IREE	ExecuTorch	ONNX Runtime	LiteRT
mobilenet_v3_small	15.9	1.97	21.3	11.7
shufflenet_v2_x1_0	21.9	4.76	39.3	9.49
mobilenet_v2	57.9	4.60	31.1	17.3
deit_tiny	—	—	102.8	38.7
efficientnet_b0	75.9	14.6	67.3	30.2
resnet18	83.9	17.4	406.1	70.3
resnet50	196.0	43.0	1012	120.7
mixer_b16	400.0	151.2	396.4	277.5
vit_b_16	649.0	542.7	547.5	367.4
vgg16	772.0	120.6	2474	338.1
deeplab_v3	8033	100.8	370.0	315.4
gemv_tower	2.90	2.87	335.9	2.73

1 thread (ms/inference, lower is better)

model	IREE	ExecuTorch	ONNX Runtime	LiteRT
mobilenet_v3_small	72.0	9.57	58.2	42.5
shufflenet_v2_x1_0	113.0	17.2	75.0	58.5
mobilenet_v2	233.0	31.0	172.4	127.1
deit_tiny	—	—	357.1	240.1
efficientnet_b0	464.0	79.3	246.3	195.5
resnet18	554.0	131.1	494.6	529.3
resnet50	1354	293.0	1145	881.4
mixer_b16	3060	943.3	2292	1941
vit_b_16	4733	1681	3302	2555
vgg16	5363	1080	4394	2593
deeplab_v3	42134	800.4	2105	1822
gemv_tower	17.1	6.27	503.3	7.81

bf16

ORT cannot load a bf16 model, ExecuTorch lowers one but XNNPACK delegates zero nodes.
IREE-only, three self-exported BF16 models:

model	threads	no data tiling	+data tiling	+data tiling +ukernels	same-weight f32
resnet50	8T	1689	1221	1196	1162
whisper-tiny-en	8T	2191	1978	1140	1580
qwen3-600m	8T	470.0	300.0	75.9	—

IREE configuration tests

Every model x every combination of --iree-opt-data-tiling (dt),
--iree-llvmcpu-enable-ukernels=all|none (uk) and
--iree-global-opt-use-im2col-for-convs (im), for fp32/fp16/int8 at both thread
counts — 288 compiles, 536 measured cells.

The two tables in this section are the 8-thread pass. The choice of configuration is
largely thread-count-independent: dt=1 uk=0 im=1 tops the same number of models at both
counts, the per-model winner is identical for 8/12 fp32, 11/12 fp16 and 7/12 int8, and
where they disagree the loser is a near-neighbour — carrying the 8-thread choice over to
one thread costs at most 1.17x (shufflenet int8), under 1.10x in 8 of 10 cases.

model (fp32)	dt=0 uk=0 im=0	dt=0 uk=0 im=1	dt=0 uk=1 im=0	dt=0 uk=1 im=1	dt=1 uk=0 im=0	dt=1 uk=0 im=1	dt=1 uk=1 im=0	dt=1 uk=1 im=1
mobilenet_v3_small	41.2	build	41.1	build	11.4	8.23	11.4	8.97
shufflenet_v2_x1_0	110.0	10.7	110.0	10.7	23.3	8.61	23.5	10.5
mobilenet_v2	255.0	25.3	255.0	25.1	52.0	34.6	51.9	38.1
deit_tiny	86.4	64.6	86.3	64.6	build	build	crash	crash
efficientnet_b0	306.0	build	306.0	build	54.5	32.3	54.5	35.4
resnet18	227.0	114.0	227.0	114.0	229.0	48.6	227.0	50.9
resnet50	1795	247.0	1800	248.0	1157	95.3	1174	107.0
mixer_b16	772.0	634.0	770.0	631.0	411.0	263.0	422.0	277.0
vit_b_16	1150	1007	1135	988.0	588.0	440.0	621.0	469.0
vgg16	1010	1310	1019	1271	879.0	518.0	890.0	526.0
deeplab_v3	2605	805.0	2609	791.0	964.0	821.0	981.0	858.0
gemv_tower	109.0	106.0	109.0	111.0	10.8	10.7	10.8	10.8

Default-vs-best across all datatypes, 8 threads (at 1 thread the gains are slightly
larger: max 12.8x fp32 / 16.6x fp16 / 19.4x int8):

model	datatype	default	best	gain
mobilenet_v3_small	fp32	11.4	8.23	1.39x
shufflenet_v2_x1_0	fp32	23.5	8.61	2.73x
mobilenet_v2	fp32	51.9	34.6	1.50x
efficientnet_b0	fp32	54.5	32.3	1.69x
resnet18	fp32	227.0	48.6	4.67x
resnet50	fp32	1174	95.3	12.32x
mixer_b16	fp32	422.0	263.0	1.60x
vit_b_16	fp32	621.0	440.0	1.41x
vgg16	fp32	890.0	518.0	1.72x
deeplab_v3	fp32	981.0	821.0	1.19x
gemv_tower	fp32	10.8	10.7	1.01x
mobilenet_v3_small	fp16	5.25	4.68	1.12x
shufflenet_v2_x1_0	fp16	9.46	5.05	1.87x
mobilenet_v2	fp16	32.1	28.5	1.13x
efficientnet_b0	fp16	30.7	23.7	1.30x
resnet18	fp16	150.0	27.3	5.49x
resnet50	fp16	761.0	48.8	15.59x
mixer_b16	fp16	313.0	140.0	2.24x
vit_b_16	fp16	403.0	189.0	2.13x
vgg16	fp16	1145	237.0	4.83x
deeplab_v3	fp16	636.0	435.0	1.46x
gemv_tower	fp16	5.37	5.31	1.01x
mobilenet_v3_small	int8	115.0	15.9	7.23x
shufflenet_v2_x1_0	int8	320.0	21.9	14.61x
mobilenet_v2	int8	541.0	57.9	9.34x
efficientnet_b0	int8	788.0	75.9	10.38x
mixer_b16	int8	629.0	400.0	1.57x
vit_b_16	int8	1112	649.0	1.71x
vgg16	int8	7859	772.0	10.18x
deeplab_v3	int8	13037	8033	1.62x
gemv_tower	int8	13.1	2.90	4.52x

What would not run on IREE

cell	what happens
DeiT-Tiny, data tiling + ukernels=none	compile: large vector sizes (32768 bytes)
DeiT-Tiny fp32/fp16 with im2col + data tiling	SIGSEGV
MobileNetV3-Small / EfficientNet-B0 / DeepLabV3, im2col without data tiling	compile: write affecting operations on global resources
VGG16 / ResNet50 / DeepLabV3 int8, data tiling + ukernels	compile: large vector sizes
int8, data tiling + ukernels (all other models)	SIGSEGV at runtime
fp16, any model, with --output=	OUT_OF_RANGE … buffer byte_length=2000 — module declares f32 result, returns f16
zvl1024b (A100), default config	7 of 15 cells SIGSEGV, enabling im2col fixes every one

Potential bugs found

• The bf16 mmt4d ukernel merged in [LLVMCPU][RISCV] Add RVV bf16 mmt4d ukernel using the Zvfbfwma widening MAC (bf16*bf16->f32) #24695 was unreachable at runtime (fixed in [Runtime][RISCV] Detect Zvfbfwma via hwprobe for the bf16 mmt4d ukernel #24767).
• Silent SIGSEGV at +zvl1024b in the default conv pipeline (e.g. mobilenet_v2 int8), adding im2col fixes the crash.
• Data-tiled int8 models crash with SIGSEGV.
• fp16 demote ABI mismatch -- running any --iree-input-demote-f32-to-f16 module with --output= fails with OUT_OF_RANGE; attempted to access an address outside of the valid buffer range (offset=0, adjusted_length=4000, end=3999, buffer byte_length=2000).

Conclusions

1. IREE is competitive on performance with other inference runtimes, but the default
   configuration is not optimal.

2. int8 compilation pipeline is being suboptimal and can use fixing. IREE is the only stack
   where quantizing costs time agains its own fp32 — 0.10–0.68x on every model tested except
   GEMV (0.07–0.45x at 1 thread) — while ExecuTorch turns the same models into a 1.4–2.3x win.
   Why quantization is a net loss on RISC-V is the question I would most like to work on.

Where I would like to go next

1. int8 codegen — figure out what exactly prevents IREE from getting good performance on int8.
2. DeepLabV3 int8 performs badly even with the best IREE compilation flags, i'd like to investigate why.
3. zvl1024b crash (though it's fairly specific to SpacemiT K3).

[dimp-pl]

@egebeysel please have a look and let me know what you think.

As a next step i can create separate issues for all of the problems i ran into when running the benchmark.

[egebeysel]

This is amazing @dimp-pl! Great work and thanks 🚀 I'll take a closer look as soon as I can, and get back to you.

Skimmed through the numbers a bit, just out of curiosity, could you take a look at the deassembly of one of your models and see if the ukernels are inlined? We currently have this issue open #24755 that I have not posted the fix for just yet, but will do so soon. I don't really understand why exactly the ukernel ones are consistently slower.

Also, as you have already noticed as well, IREE currently does data-tiling to convolutions via the im2col transformation to matmuls. Some of your results also show that the non-data-tiled configurations actually run better than with data-tiling. That most probably boils down to amortizing the costs of the packing/unpacking AND the inherent overhead of the im2col transformation itself, which generally tends to tip the scale towards not using data-tiling, at least the way we have it currently.

That's why, to enable even more efficient lowerings, we are actively working on a native data-tiling path for convolution operations: #24710 . We're expecting better performance with this approach, stay tuned! Currently, this was enabled on x64 and AArch64 but this might be a good direction to enable this for RISC-V as well :)

I'll get back to you as soon as I can, but I'm OOO until Tuesday, so bear with me. In the meanwhile, feel free to take a look at the potential bugs you might've found and please file issues, I'm sure someone will assist you if necessary!

[egebeysel]

Also, just by looking at the models, maybe we could also throw in some language models (inherently matmul-heavy) to see how they perform?

[NoumanAmir657]

Hi @dimp-pl, this is very comprehensive. One thing that I want to confirm is that currently, you are either enabling all ukernels or enabling none. For RISC-V, pack and unpack ukernels don't exist so they fall back to scalar code (correct me if I am wrong here). Maybe you can try with enabling only mmt4d ukernel because that would make the pack and unpack operations fall to generic vectorization which can vectorize them.
Another thing is that usually using iree-dispatch-creation-data-tiling is able to fuse to pack with mmt4d, so you might also want to try that flag.

[dimp-pl]

@NoumanAmir657 good idea, thanks, let me try that.

[egebeysel]

JFYI: --iree-opt-data-tiling currently defaults to dispatch creation data tiling if I'm not mistaken.