rten-examples

rten-examples

This crate contains example projects showing how to convert and run models for common tasks across various modalities. See Example descriptions for a summary of what each example does.

ONNX format support

The example instructions use models in .onnx format. This is only supported by rten v0.23 and later. To run examples with earlier versions of rten, you will need to convert the models to .rten format first:

$ pip install rten-convert
$ rten-convert model.onnx

This will create a model.rten file in the same directory as the model.

Running an example

Each example has a main function with a comment above it describing the steps to fetch the ONNX model and run the example.

The general steps to run an example are:

1. Download the ONNX model. These usually come from Hugging Face, or ONNX format models pre-created by the model authors or community members.

2. Run the example from the rten-examples directory using:

   $ cargo run -r --bin <example_name> -- <...args>

   Where ...args refers to the example-specific arguments, such as model path(s), tokenizer configuration and input data.

   Note the -r flag to create a release build. This is required as the examples will run very slowly in debug builds.

   The syntax and flags for an individual example can be displayed using its --help command:

   $ cargo run -r --bin <example_name> -- --help

   Note the -- before --help. Without this cargo will print its own help info.

   You can also run examples from the root of rten repository by specifying the rten-examples package name:

   $ cargo run -p rten-examples -r --bin <example_name> -- <...args>

Reference implementations

Some of the examples have reference implementations in Python using PyTorch and Transformers. These are found in src/{example_name}_reference.py and enable comparison of RTen outputs with the original models.

Example descriptions

The examples have been chosen to cover common tasks and popular models.

Vision

• clip - Match images against text descriptions using CLIP
• imagenet - Classification of images using models trained on ImageNet. This example works with a wide variety of models, such as ResNet, MobileNet, ConvNeXt, ViT.
• deeplab - Semantic segmentation of images using DeepLabv3
• depth_anything - Monocular depth estimation using Depth Anything
• detr - Object detection using DETR or RT-DETR (Real-time DETR).
• distilvit - Image captioning using Mozilla's DistilViT
• nougat - Extract text from academic PDFs as Markdown using Nougat
• rmbg - Background removal using BRIA Background Removal
• segment_anything - Image segmentation using Segment Anything
• trocr - Recognize text using TrOCR
• yolo - Object detection using Ultralytics YOLO

Text

• bert_qa - Extractive question answering using BERT-based models which have been fine-tuned on the SQuAD dataset
• gpt2 - Text generation using the GPT-2 language model.
• jina_similarity - Sentence similarity using vector embeddings of sentences
• llama - Chatbot using Llama 3
• modernbert - Masked word prediction using ModernBERT. Also works with the base version of the original BERT model.
• qwen2_chat - Chatbot using Qwen2. Also works with some other chat models that use the same prompt format such as SmolLM.

Audio

• byt5_g2p - Convert text to phonemes using ByT5. The phonemes can be used as inputs for the piper example.
• kokoro - Text-to-speech using Kokoro.
• piper - Text-to-speech using Piper models
• silero - Speech detection using Silero VAD
• wav2vec2 - Speech recognition of .wav audio using wav2vec2
• whisper - Speech recognition of .wav audio using OpenAI's Whisper