Lab03

Lab 3

---

Lab Notebook

1. Complex Yet Simple Training Pipeline
2. Inference Optimization And TTA

The exercises from these notebook are a good preparation for homework 2, and can help you acquire MLOps or Data Science skills.

Bonus points

You will get bonus points if you do all 4 exercises from "Complex Yet Simple Training Pipeline" and submit them until Lab 4.
You will get bonus points if you do all MLOps exercises from "Inference Optimization And TTA" and submit them until Lab 5.

---

Homework 2: https://www.kaggle.com/t/5fe1947f27b743beb65e005fd709cf79

---

For self-study (for students who want to pass):

• Skim over the 2 references, LeCun (98) and Keskar (2017).
• Watch the videos for AlexNet and ResNet.
• Convolutions:
  • But what is a convolution? (convolution example; convolutions in image processing; convolutions and polynomial multiplication; FFT)
  • CNN Explainer (convolutions applied in neural networks)
• Hyperparameter tuning / experiment tracking:
  • Tensorboard: https://pytorch.org/docs/stable/tensorboard.html
  • Weights and Biases: https://docs.wandb.ai/guides/integrations/pytorch

---

Advanced (for students who want to learn more):

• Learn how to choose hyperparameters and the influence of batch size:
  • Gradient Based Learning Applied to Document Recognition (LeCun, 98)
  • On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (Keskar, 2017)
• Read the foundational CNN papers:
  • AlexNet:
    • https://www.cs.toronto.edu/~hinton/absps/imagenet.pdf
    • [Classic] ImageNet Classification with Deep Convolutional Neural Networks
  • ResNet:
    • https://arxiv.org/abs/1512.03385
    • [Classic] Deep Residual Learning for Image Recognition