AI Hardware

[TOC]

Metrics

	Full Name	Measures	Typical Use Case
FLOPS	Floating Point Operations Per Second	Floating-point performance	Scientific computing, GPUs, HPC
TOPS	Tera Operations Per Second	AI-specific operations	AI inference, NPUs, accelerators
MIPS	Million Instructions Per Second	General instruction throughput	CPUs, embedded systems
DMIPS	Dhrystone MIPS	Real-world CPU performance	Benchmarking CPUs and MCUs
Hash/s	Hashes Per Second	Cryptographic hashing rate	Mining, security workloads

Precision Format

Precision Type	Typical Use Case	Performance (TOPS)	Accuracy Impact	Common in Edge AI?
INT8 (8-bit integer)	Vision inference, object detection, speech recognition	Highest	Slight drop in accuracy vs FP32	Yes: favored for speed & efficiency
FP16 (16-bit floating point)	Real-time processing where some precision is needed	High	Small accuracy loss	Sometimes: balance of speed & precision
FP32 (32-bit floating point)	Model training, scientific computing, high-precision inference	Lower	Full precision	Rare: too slow/power-hungry for most edge use

FLOPS (Floating Point Operations Per Second)

FLOPS (Floating-Point Operations per Second) is a measure of a computer’s ability to perform arithmetic calculations on real numbers.

FLOPS Calculation Formula: $$ FLOPS = \frac{\text{Number of Floating-Point Operations}}{\text{Execution Time (in seconds)}} $$ MFLOPS (Millions of FLOPS) Calculation Formula: $$ MFLOPS = \frac{\text{Number of Floating-Point Operations}}{\text{Execution Time (in seconds)} \times 10^{6}} $$ GFLOPS (Billions of FLOPS) Calculation Formula: $$ GFLOPS = \frac{\text{Number of Floating-Point Operations}}{\text{Execution Time (in seconds)} \times 10^{9}} $$ TFLOPS (Trillions of FLOPS) Calculation Formula: $$ TFLOPS = \frac{\text{Number of Floating-Point Operations}}{\text{Execution Time (in seconds)} \times 10^{12}} $$ PFLOPS (Quadrillions of FLOPS) Calculation Formula: $$ PFLOPS = \frac{\text{Number of Floating-Point Operations}}{\text{Execution Time (in seconds)} \times 10^{15}} $$ Estimate CPU/GPU/etc Formula: $$ FLOPS = \text{Core Count} \times \text{Frequency} \times \text{Instructions per Cycle} \times \text{FLOPs per Instruction} $$ For Example:

A CPU with 4 cores at 3.5 GHz, 4 IPC, and 1 FLOP per instruction: $$ 4 \times 3.5,GHz \times 4 = 56,GFLOPS $$

Factors Affecting FLOPS Performance:

Processor Architecture: The design of the CPU or GPU determines how efficiently it performs floating-point operations. Modern processors have specialized units called Floating Point Units (FPUs) for these tasks.
Clock Speed (GHz): Higher clock speeds allow more instructions per second, improving FLOPS—though architecture and parallelism also matter.
Parallelism: Systems with multiple cores or threads can execute many operations simultaneously. GPUs, for instance, excel in parallel computation, achieving high FLOPS rates.
Memory Bandwidth: If data cannot be supplied to the processor fast enough, performance drops. High bandwidth memory allows continuous data flow, improving FLOPS.
Algorithm Efficiency: Well-optimized algorithms require fewer operations to achieve the same result, thus improving effective FLOPS.

TOPS (Tera Operations Per Second)

TOPS measures trillion operations per second and is widely used in AI accelerators, NPUs, and ML inference engines.

TOPS (Tera Operations Per Second) Calculation Formula: $$ TOPS = \text{Clock Frequency} \times \text{Instructions per Cycle} \times \text{Ops per Instruction} $$

MIPS (Million Instructions Per Second)

MIPS measures the number of instructions a CPU can execute in one second, indicating its processing speed. A higher MIPS generally means the processor can handle more tasks, but it does not always reflect real-world performance.

Calculate MIPS(Million Instructions Per Second) Formula: $$ \text{MIPS} = \frac{\text{Number of Instructions}}{\text{Execution time} \times 10^{6}} $$ Calculate IPS(Instructions per second) Formula: $$ \text{Instructions per second (IPS)} = \frac{\text{CPU Clock Speed}}{\text{CPI}} $$ For Example:

A CPU executing 500,000 instructions per second achieves: $$ MIPS = \frac{500,000}{10^{6}} = 0.5 $$

DMIPS (Dhrystone MIPS)

DMIPS is based on the Dhrystone benchmark, providing a more realistic measure of CPU performance under typical workloads.

Calculate DMIPS (Dhrystone MIPS) Formula: $$ DMIPS = \frac{\text{Dhrystone Instruction Count}}{10^{6}} $$ For Example:

A CPU executing 800,000 Dhrystone operations per second achieves: $$ DMIPS = \frac{800,000}{10^{6}} = 0.8 $$

Hash/s (Hashes Per Second)

Used primarily in cryptography and blockchain mining, Hash/s quantifies how many hash calculations a device can perform per second.

For Example:

If a system performs 100,000 SHA-256 hashes per second:

Hash Rate = 100,000

Central Processing Units (CPUs)

In a computer system, processing and control are handled by a crucial component responsible for executing instructions and managing operations.

Control Unit: The control unit manages the CPU by sending signals like clock, hold, and reset to its parts. It ensures all components work together to complete tasks.
Arithmetic and Logic Unit (ALU): The ALU handles arithmetic tasks (like addition, subtraction, multiplication, division) and logical tasks (like AND, OR, comparisons). It uses addition for all calculations.
Memory Unit: The memory unit stores data and instructions. Older CPUs used registers, but modern ones also have fast cache memory. The CPU fetches data from RAM, ROM, or hard disks and stores it in registers or cache during tasks.

Functions

The functions of the CPU involve processing instructions from programs and controlling all operations within the computer. This is carried out through a sequence known as the Fetch-Decode-Execute-Store cycle:

Fetch: The CPU retrieves the instruction from main memory (RAM).
Decode: The Control Unit interprets the fetched instruction to determine the required operation.
Execute: The CPU performs the operation using the appropriate hardware components, such as the ALU.
Store: The result of the executed instruction is written back to memory or a register.

CPU Make Computer Faster

Modern CPUs are designed to be super-efficient. Here are a few ways they speed things up:

Multiple Cores: Many CPUs have multiple cores, which are like mini-CPUs that can work on different tasks at the same time. It’s like having several chefs in the kitchen instead of one.
Faster Clocks: The clock speed (measured in GHz, like 3.5 GHz) determines how many instructions the CPU can handle per second.
Bigger Cache: More cache means the CPU can store more data close by, reducing wait times.
Pipelining: This lets the CPU start working on the next instruction before finishing the current one, like a factory line.

Graphics Processing Units (GPUs)

GPUs are specialized hardware designed to accelerate computations that involve parallel processing, making them crucial for deep learning and other computationally intensive ML tasks. They are particularly effective in handling large-scale matrix operations, which are common in training neural networks.

GPU Workflow

At the top level, a GPU chip is made up of many Streaming Multiprocessors (SMs). Think of SMs as mini parallel engines replicated across the chip. Instead of one big brain, you get dozens of smaller ones working simultaneously.

SM

Inside each SM:

A Warp Scheduler decides which group of threads (a warp) runs next.
Dozens of CUDA Cores execute instructions in parallel.
A Register File stores thread-local data at ultra-low latency.
Load/Store units move data between registers and memory.
Texture units handle specialized memory operations.
L1 Cache provides fast, on-SM data access.

Each SM works independently, but they’re connected through an on-chip interconnect. Below that sits the L2 Cache, shared across all SMs. This is the coordination layer. If one SM misses in L1, it checks L2 before going to global memory.

Memory Controller

Then come the Memory Controllers, which interface with Global Memory. This is where things get interesting:

Extremely high bandwidth
Much higher latency than on-chip memory

Tensor Processing Units (TPUs)

The TPU includes the following computational resources:

Matrix Multiplier Unit (MXU): 65, 536 8-bit multiply-and-add units for matrix operations.
Unified Buffer (UB): 24MB of SRAM that works as registers
Activation Unit (AU): Hardwired activation functions.

There are 5 major high-level instruction sets devised to control how the above resources work:

TPU Instruction	Function
Read_Host_Memory	Read data from memory
Read_Weights	Read weights from memory
MatrixMultiply/Convolve	Multiply or convolve with the data and weights, accumulate the results
Activate	Apply activation functions
Write_Host_Memory	Write result to memory

Field-Programmable Gate Arrays (FPGAs)

FPGA stands for Field Programmable Gate Array, which is an IC that can be programmed to perform a customized operation for a specific application. They have thousands of gates. In the field of VLSI FPGAs have been very popular. Languages such as VHDL and Verilog are used to write the code for FPGA programming.

Architecture

Types

Types of FPGA Based on their applications, FPGAs are classified as :

Low-End FPGAs: They consume less power than the other two and are less complex as no of gates is less.
Mid-Range FPGAs: They consume more power than low-end FPGAs and have a larger number of gates, so more complex. They provide a balance between performance and cost.
High-End FPGAs: They have a large gate density, so are more complex than mid-range. Their performance is better than low-end and mid-range FPGAs, but some High-End FPGAs.

Application-Specific Integrated Circuits (ASICs)

ASIC stands for Application Specific Integrated Circuit. It is specially built for a specific application or purpose. If compared to any other device, ASIC has improved speed. Basically, it is an integrated circuit that's been specified for one specific purpose and is not software programmable to perform a wide variety of different tasks.

Types

Memory and Storage

Random Access Memory (RAM)

RAM is crucial for the smooth operation of any computing system. In machine learning, it helps in loading and processing datasets, managing intermediate computations, and running multiple processes simultaneously.

Storage

Storage is vital for housing datasets, models, and software. Fast and ample storage helps in quickly loading data and saving model checkpoints.

Networking

Network speed and reliability are important for downloading datasets, accessing cloud services, and collaborating on distributed ML tasks.

Motherboard

The motherboard connects all hardware components and determines compatibility and expandability.

Cooling System

Effective cooling is crucial for maintaining optimal performance and preventing overheating of high-performance components like GPUs and CPUs.

Power Supply Unit (PSU)

The PSU provides power to all components and must be able to supply sufficient wattage for your entire system.

Summary

Different Hardware Requirements for AI

Hardware Type	Strengths	Limitations	Best Use Cases
CPUs (Central Processing Units)	+ Versatility: CPUs are general-purpose and can handle a wide variety of tasks beyond AI. + Sequential Processing: Good for any sort of work that either cannot be done in parallel well or has a very high single-thread performance requirement. + Affordable: In general, more widely available and often cheaper than specialized hardware.	+ Parallel Processing: CPUs are less efficient than GPUs for tasks demanding massive parallelism, such as deep learning. + Performance: AI tasks, especially the training of big models, run more slowly on a CPU compared to a GPU or TPU.	+ General-purpose computing. + Data preprocessing.
GPUs (Graphics Processing Units)	+ Parallelism: Parallelism makes it a great support for parallel processing, making it ideal for training deep learning models and running simulations. + Optimization: Supported by most of the AI frameworks, such as TensorFlow and PyTorch, through optimized libraries like NVIDIA's CUDA. + Scalability: Easily scalable to multiple GPUs in order to handle huge AI workloads.	+ Energy consumption: being power-hungry in nature, GPUs exhibit increased energy costs. + Cost: High-performance GPUs are usually quite costly, with the cost multiplying in large deployments.	+ Training deep learning models. + Image and video processing. + Running large-scale simulations.
TPUs (Tensor Processing Units)	+ AI Optimization: Specially designed for workloads of artificial intelligence, particularly for tensor operations used in deep learning. + Performance: Outperforms some performance benchmarks by GPUs in certain tasks, especially within the Google TensorFlow ecosystem. + Efficiency: Noticeably more power-efficient than GPUs for AI-type workloads.	+ Flexibility: Less versatile than GPUs since TPUs are optimized for very specific types of AI tasks and may not use a wide range of frameworks. + Accessibility: It is mainly available through Google Cloud, which may limit its use outside this environment.	+ TensorFlow-based deep learning tasks. + Large-scale AI model training. + Cloud-based AI deployments.
FPGAs (Field-Programmable Gate Arrays)	+ Customization: They can be reprogrammed, providing a balance between good performance and flexibility in optimization for specific AI tasks. + Reduced Latency: This is significant in that FPGAs can offer reduced latency compared to CPUs or GPUs and, therefore, prove ideal for real-time AI applications. + Energy Efficiency: Often — but not always — more energy efficient than CPUs and GPUs for specific tasks.	+ Programming Complexity: Specialized knowledge in programming and optimization is required, which then becomes a barrier to wider usage. + Performance: Although flexible, the performance of FPGAs does not reach that of raw performance from GPUs or TPUs in large-scale AI tasks.	+ Real-time AI applications. + Low-latency requirements. + Specialized, customizable AI tasks.
ASICs (Application-Specific Integrated Circuits)	+ Performance: Best performance provided for specific tasks, as they are built to be application-specific. + Power efficiency: Highly power-efficient for the tasks at hand.	+ Cost and Development Time: As ASICs are costly and take a lot of time in their development, they are recommended for high-volume, special-purpose applications. + Lack of flexibility: Once designed, the ASICs cannot be reprogrammed, and hence they find applications among only those targeted.	+ High-volume, specialized AI applications. + Edge computing with specific AI functions. + Large-scale data centers.

TPU vs GPU vs CPU

Feature	CPUs	GPUs	TPUs
Architecture	CPUs are general-purpose processors optimized for sequential and branch-heavy tasks.	GPUs are designed for rendering graphics and handling parallel tasks.	TPUs are specialized hardware developed by Google for AI tasks.
Flexibility	CPUs are highly flexible and can run almost any operating system and software workload.	GPUs are versatile and used in gaming, video rendering, and AI.	TPUs are optimized specifically for tensor operations in AI.
Performance	CPUs perform well on control-intensive tasks and workloads with strong single-thread dependence.	GPUs excel in tasks requiring high precision and flexibility.	TPUs provide superior performance for inferencing tasks.
Energy Efficiency	CPUs are power-efficient for general-purpose workloads but less efficient for large-scale parallel AI compute.	GPUs are energy-efficient but can consume significant power under load.	TPUs are designed for high efficiency in specific AI tasks.
Manufacturers	Leading CPU manufacturers include Intel, AMD, and ARM-based vendors such as Apple.	Leading GPU manufacturers include NVIDIA and AMD.	Google is the primary developer and manufacturer of TPUs.
Use Case	CPUs are widely used for system control, data preprocessing, and mixed general workloads.	GPUs are widely used in training complex AI models.	TPUs are optimized for real-time inference and TensorFlow tasks.
Latency	CPUs can provide low latency for control logic and small-batch inference tasks.	GPUs can have higher latency in real-time tasks compared to TPUs.	TPUs achieve low latency, typically around two seconds.
Scalability	CPUs scale well across multi-core and multi-socket servers, but AI scaling is slower than GPUs/TPUs.	GPUs scale well for extensive computational tasks and large datasets.	TPUs are highly scalable within Google's ecosystem and cloud services.
Specialization	CPUs are least specialized and designed to support a wide variety of computing tasks.	GPUs are general-purpose and cater to a broad range of applications.	TPUs are highly specialized for deep learning and AI operations.
Deployment	CPUs are deployed universally in personal devices, edge systems, servers, and cloud platforms.	GPUs are commonly used in personal computers, data centers, and cloud services.	TPUs are primarily deployed within Google's infrastructure and cloud services.