Edge AI Handbook: A Practical Approach to AI Application Development

[stevenhoenisch]

The Edge AI Handbook: A Practical Approach illustrates how to develop AI applications. The handbook spans system selection, system setup, system profiling, accuracy optimization, performance optimization, and deployment. Sample applications serve as implementation references, and you can use the handbook to set up and optimize such systems as Deep Learning Streamer, Geti™, and OpenVINO™. Benchmark results that cover AI tasks performed at the edge help you make decisions about initial system selection and setup, including for both bare-metal and containerization methods.

The handbook also covers other topics important for making decisions about system selection, setup, and tuning. System profiling, for instance, becomes critical to measure resource allocation, especially for model benchmarking, and that's addressed in the handbook.

• Accuracy optimization and performance optimization sections share tips on tuning.

• The system setup section includes steps for installing training tools to improve model accuracy.

• The deployment section provides both streaming and serving inference pipeline methods.

• Information about real-time processing at the edge, where applications require an instantaneous response, making latency, rather than throughput, a key performance indicator.

• See how an integrated GPU (iGPU) performs with edge applications like OpenVINO.

Some of the key topics discussed in the Edge AI Handbook include the following.

🎢 A Practical Discussion of Processor SKU Selection

Edge AI applications span many industries:

• Robotics
• Manufacturing
• Retail
• Education
• Smart cities and infrastructure
• Healthcare and health and life sciences

As such, the handbook reflects the variety of systems across the entry, light, medium, and heavy AI categories. For example, the systems for entry AI are normally sub 15 watts that can only process one camera up to 30 frames per second (FPS). For light AI systems, the power range is 15 to 65 watts without dGPU. The light AI with additional 50, 100, or 200 watts dGPU falls into medium AI category. Finally, the heavy AI systems have server processors with over 65 watts CPU and multiple 100s of watts of dGPUs. The section on processor SKU selection will help give you a rough idea of which system is suitable for your application using OpenVINO™ 2025 Performance Benchmarks Results.

🎯 Geti Model Selection and Optimization for Accuracy

Geti software eases laborious data upload, labeling, model training, retraining, and optimization tasks to streamline the development of computer vision models. You can rapidly build vision models for various processes, including detecting defective parts in a production line, reducing downtime on the factory floor, or automating inventory management.

The Geti™ application supports multiple models for each task, and the Edge AI Handbook recommends certain models for better accuracy, faster speed, or a balance of the two.

🎬 To find out how to build computer vision models in a fraction of the time and with less data, in effect turbocharging the entire vision AI lifecycle, including training on Intel® Arc™ GPUs, check out the following series of videos on YouTube:

• Why Geti? Build Vision AI Without Being an Expert

• Geti in Full Flow: Installation and Deployment Made Simple

• How AI Training Works: Deep Learning and Fine-Tuning Explained

• The Complete Geti Playlist, which gives you, in effect, a full course on Geti and building computer vision models.

⚟ OpenVINO Toolkit Model Conversion

Converting models from TensorFlow, PyTorch, or other deep learning framework models to OpenVINO™ Intermediate Representation (IR) models has the benefit of running the models with the best performance on Intel® processors, including CPU, iGPU, NPU, and dGPU. After converting the model, it is recommended to run OpenVINO™ benchmark_app to obtain optimal model parameters. For details, see the Edge AI Handbook.

🎬 Here's a selection of videos on OpenVINO™ to supplement the information in the handbook:

• OpenVINO C++

• OpenVINO GenAI pipelines C++

• OpenVINO model hub

• OpenVINO speculative decoding

• pyTorch 2.7 on Intel GPU

🕹️ Illustrating the Power of Panther Lake with a Video QnA App

The Intel® Core™ Ultra Processor Series 3 (Panther Lake-H) iGPU has about 60% more capability than its predecessor, the Intel® Core™ Ultra Processor Series 2 (Arrow Lake-H) iGPU. With additional compute power, the Edge AI Handbook uses video input for the VQA application rather than an image. Using a static image, the AI can only identify whether a person is holding an item. The benefit of using a video is that the AI can now identify whether a person is picking up, putting down, or just carrying an item. Instead of running the VQA around the clock, a developer can use object detection running on a low-power NPU to trigger the video QnA application running on a GPU.

Check out the Edge AI Handbook for details. 👀 🏗️ 🚀

[biapalmeiro]

Very insightful overview. Thank you!