Outline of ViT - Common Vision Transformers

[SuryaKrishna02]

Hey Everyone,
Following is the Outline for the ViT - Common Vision Transformers Chapter:

Basic Format of every section:

• Overview and Rational (including some model history)
• Advantages of Architecture
• Implementation Details / Key Features
• Case Studies and/or Live Application (PyTorch)

1. Introduction

• Background on Transformers
• Motivation for using Transformers-based architectures over CNN.
• Overview of common vision transformers: SWIN, DETR, CvT, ... etc.

2. SWIN Transformer. Contributor: @klyap

• Overview and Rationale
• Advantages of Architecture: efficiency, scalability, enhanced representation
• Implementation Details / Key Features: Window-based Self-attention, Hierarchical Representation, and Relative Position Bias, Patch Merging
• Case Study: Applications in classification, object detection, and segmentation.

3. DEtection TRansformer. Contributor: @alvanli

• Overview and Rationale
• Advantages of Architecture: Architecture simplicity, global reasoning, and end-to-end training process
• Implementation Details / Key Features: Encoder-Decoder Architecture, Bipartite Matching, and Set-based Global Loss Function
• Case Study: Applications in class predictions

4. CvT. Contributor: @SuryaKrishna02

• Overview and Rationale
• Advantages of Architecture: Advantages and Challenges
• Implementation Details / Key Features: Deep Dive of Architecture
• Case Study: Applications and Use Cases
• Application: Hands-on using HuggingFace CvT for Image Classification

6. MobileViT. Contributor: @Mkrolick

• Overview and Rationale:
  • Evolution of architecture: DeepLabv3, MobileNetv2, Apple Research, ViT, MobileViT v1
  • Include Brief overview of MobileViT v1 - MobileViT block and rationale behind architecture
• Advantages of Architecture:
  • Reduced MHA Complexity O(k^2) -> O(k), Less parameters, Lightweight Architecture (Less computationally expensive operations - IE batch-wise matrix multiplication)
• Implementation Details / Key Features: Multihead attention -> Self Separable Attention, Convolutions & Pooling, MobileNetv2 Blocks, Downsampling
• Case Study: Live implementation of Mobile ViT's potential on Mobile Devices
• Application: Building a lightweight classifier

7. DiNAT: Contributor: @alanahmet

• Overview and Rationale:
  • Introduction to DiNAT Transformers
  • Comparing DiNAT with Other Transformer Variants
• Advantages of Architecture:
  • Exploring the Core Concepts of DiNAT: Neighborhood Attention and Global Context
• Implementation Details / Key Features:
  • Configuring DiNAT Models: A Detailed Look at Parameters and Options
  • Implementation of Neighborhood Attention and Global Context in Pytorch
  • Implementing DiNAT: Step-by-Step Guide to Building DiNAT Models
  • Hands-on: DiNAT Model Initialization and Training
• Case Study:
  • DiNAT in Object Detection: Enhancing Performance with DiNAT-Based Detectors
  • Semantic Segmentation with DiNAT: Achieving State-of-the-Art Results
  • Industry Use Cases: DiNAT in Healthcare, Autonomous Vehicles, and More
• Application:
  • Building an Image Classifier using DiNAT
  • Enhancing Object Detection with DiNAT Transformers
  • Capstone Project: Developing a DiNAT-Powered Vision Application

Regards,
Common Vision Transformers Team.