CI Test Benchmarking Suite

Overview

This page documents the current state of the FINN+ CI suite. It is executed on our local infrastructure via GitLab CI pipelines based on a mirror of this repository.

Automatic testing of the FINN+ compiler is based on three pillars:

• PyTest unit test suite
• Regression tests on example models (see Example models documentation)
• Micro-benchmarks of individual back-end components

Planned improvements/extensions are tracked in this issue.

What runs when?

Suite (regression test suites defined in ci/cfg/)	Push to dev/main	PR to dev	Only manually
Full PyTest suite on Vitis/Vivado 2024.2	x	x
Full PyTest suite on Vitis/Vivado 2022.2	x
Basic set of regression tests (regression_basic.yml)	x	x
Extended set of regression tests (regression_extended.yml)	x
Regression tests for Transformer models (regression_transformer.yml)	x
Basic set of micro-benchmarks (microbenchmark_basic.yml)	x
Extended set of micro-benchmarks (TODO)			x
Live FIFO-Sizing for end2end models and synthetic non-linear models (live_fifosizing.yml)			x
Any custom experiment config that is not defined within the repository (ci/cfg/)			x

Regression Test Features

Suite	Model	Dataset	Target	Flow verification	On-board validation	On-board measurement	Issues
basic	VGG-10*	RadioML 2018	RFSoC2x2	No	No	Yes
basic	MNv1*	ImageNet 2012	RFSoC2x2	No	No	Yes
basic	Cybersecurity (MLP)*	UNSW-NB15	RFSoC2x2	No	No	Yes
extended	BNN-PYNQ tfc* (3 variants)	MNIST	RFSoC2x2	No	No	Yes
extended	BNN-PYNQ cnv* (3 variants)	CIFAR-10	RFSoC2x2	No	No	Yes
extended	GTSRB (cnv-w1a1)*	GTSRB	RFSoC2x2	No	No	Yes
extended	KWS (MLP)*	Google Speech Commands v2	RFSoC2x2	No	No	Yes	Broken (see #101)
extended	RN-50*	ImageNet 2012	U250	No	No	No	Disabled (to be replaced with new generic ResNet flow)
transformer	Transformer dummy (2 variants)	N/A	RFSoC2x2	Yes	No	Yes
transformer	New Transformer dummy (1 variant)	N/A	RFSoC2x2	Yes	No	Yes	Broken (new streamlining WIP)
transformer	Transformer GPT (4 variants)	TinyStories	U250	Yes	No	No	See #84
transformer	Transformer RadioML (5 variants)	RadioML 2018	RFSoC2x2	Yes	No	Yes	See #84
transformer	Conformer RadioML (1 variant)	RadioML 2018	U250	Yes	No	No

* based on finn-examples