linear_algebra_review.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "602ad268",
   "metadata": {},
   "source": [
    "# Linear Algebra Review"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92387722",
   "metadata": {},
   "source": [
    "This page reviews some basic linear algebra for machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ebdd849",
   "metadata": {},
   "source": [
    "## Overview\n",
    "- Vectors\n",
    "    - definition\n",
    "    - norm of a vector\n",
    "    - operations\n",
    "    - dot products\n",
    "- Matrices\n",
    "    - definition\n",
    "    - operations\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab38a7f3",
   "metadata": {},
   "source": [
    "## Vectors\n",
    "\n",
    "Tuple of real numbers.\n",
    "\n",
    "- $V=[V_1, V_2, ... V_d]$\n",
    "\n",
    "- $V_1, V_2,,$ are components of vector $V$.\n",
    "\n",
    "This vector has $d$ members; hence, this vector's dimension is $d$.\n",
    "\n",
    "Dimension of the vector is represented as follows:\n",
    "\n",
    "- $V  \\in \\mathbb{R}^d$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee59e8e2",
   "metadata": {},
   "source": [
    "###  Example vector, d=2\n",
    "\n",
    "<div>\n",
    "<img src=\"figures/vector_ex.png\" width=\"300\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "An example vector $a=[2,3]$.\n",
    "\n",
    "We use $x$ and $y$ as convention, but we can draw and denote as $x_1$ and $x_2$ for 2 dimensional vector space.\n",
    "\n",
    "A vector has magnitude and direction.\n",
    "\n",
    "Length of this vector is $\\sqrt{2^2 + 3^2}$\n",
    "\n",
    "$\\tan \\theta$ of the vector is $\\frac{3}{2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2a9f8de",
   "metadata": {},
   "source": [
    "### Norm of a vector\n",
    "\n",
    "*Norm* is a length or size of the vector in the vector space.\n",
    "\n",
    "\n",
    "- $V=[V_1, V_2, ... V_d]$\n",
    "\n",
    "- $V_1, V_2,,$ are components of vector $V$.\n",
    "\n",
    "- Norm is shown as double bar, $||V||$, and it is a scalar value, i.e $||V|| \\in \\mathbb{R}$\n",
    "\n",
    "There are different types of norms. We will introduce $l^2$ norm (read l 2 norm) also known as euclidean norm.\n",
    "\n",
    "- $||V||_2 = \\sqrt{V_1^2 + V_2^2 + ... + V_d^2} = \\sqrt{\\sum_{i=1}^d V_i^2}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26b662a2",
   "metadata": {},
   "source": [
    "### Operations of vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe19402",
   "metadata": {},
   "source": [
    "#### Element-wise operations\n",
    "\n",
    "Apply operations (sum, subtractions, ...) element by element of vectors.\n",
    "\n",
    "- $a, b \\in \\mathbb R^d$\n",
    "- $c = a + b$\n",
    "\n",
    "The result $c$ is also $d$ dimension, $c \\in \\mathbb R ^d$\n",
    "\n",
    "\n",
    "The following shows the element-wise operation.\n",
    "\n",
    "- $c_i = a_i + b_i$ for $i=0, .. d$\n",
    "- $c = [a_1+b_1, a_2+b_2, ... a_d+b_d]$\n",
    "\n",
    "\n",
    "The following is an example of `numpy` code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cd1f4f08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 5 7]\n",
      "[3 5 7]\n",
      "[-1 -1 -1]\n",
      "[0.5        0.66666667 0.75      ]\n"
     ]
    }
   ],
   "source": [
    "# elementwise operstion example\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "y = np.array([1,2,3])\n",
    "x = np.array([2,3,4])\n",
    "\n",
    "print(y + x) # [3, 5, 7]\n",
    "print(y + x) # [3, 5, 7]\n",
    "print(y - x) # [-1, -1, -1]\n",
    "print(y / x) # [.5, .67, .75]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7d40df",
   "metadata": {},
   "source": [
    "#### Scalar multiplications\n",
    "\n",
    "Apply scalar value to all vector elements.\n",
    "\n",
    "- $V \\in \\mathbb R^d$, $k \\in \\mathbb R$\n",
    "- $c = k V$\n",
    "\n",
    "The result $c$ is also $d$ dimension, $c \\in \\mathbb R ^d$\n",
    "\n",
    "\n",
    "The following shows the element-wise operation.\n",
    "\n",
    "- $c_i = k V_i$ for $i=0, .. d$\n",
    "- $c = [kV_1, kV_2, ... kV_d]$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d8499a5",
   "metadata": {},
   "source": [
    "#### Dot product\n",
    "\n",
    "Dot product is ths sum of the products of each element of two vectors. The output of the dot product of two vectors are scalar.\n",
    "\n",
    "- $a, b \\in \\mathbb R^d$\n",
    "- $c = a \\cdot b$\n",
    "\n",
    "The output of the dot product of two vectors are scalar, $c \\in \\mathbb R$\n",
    "\n",
    "\n",
    "The following shows the dot product.\n",
    "\n",
    "- $c = \\sum_{i=1}^d a_i b_i$\n",
    "\n",
    "- $c = a_1b_1 +  a_2b_2, ..., + a_db_d$\n",
    "\n",
    "\n",
    "\n",
    "When divided by magnitude of two vectors, then it is cosine similarity, which is a way to measure how two vectors are close to each other.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd33a6f3",
   "metadata": {},
   "source": [
    "## Matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faf18f1b",
   "metadata": {},
   "source": [
    "Matrices can be defined as a tuple of vectors.\n",
    "\n",
    "We have the $m$ vectors.\n",
    "\n",
    "- $P_1, P_2, ..., P_m \\in \\mathbb R^n$\n",
    "\n",
    "\n",
    "We can define a matrix with the above vectors in the dimension of $m$ row, $n$ columns.\n",
    "\n",
    "$P$ = $\\begin{bmatrix}\n",
    "  <- P_1 -> \\\\ \n",
    "  <- P_2 -> \\\\\n",
    "  ... \\\\\n",
    "  <- P_m ->\n",
    "\\end{bmatrix}$\n",
    "\n",
    "- $P \\in \\mathbb R^{mxn}$, here $m$ is number of row, $n$ number of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d0744ffc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mat a:\n",
      " [[1 2 3]\n",
      " [4 5 6]]\n",
      "dim of mat a: (2, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# matrix example\n",
    "\n",
    "a = np.array([\n",
    " [1,2,3],\n",
    " [4,5,6]\n",
    "])\n",
    "\n",
    "print(\"mat a:\\n {}\".format(a))\n",
    "print(\"dim of mat a: {}\".format(a.shape))\n",
    "\n",
    "a.shape == (2,3)\n",
    "b = np.array([\n",
    " [1,2,3]\n",
    "])\n",
    "\n",
    "b.shape == (1,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c5a808",
   "metadata": {},
   "source": [
    "### Matrix - vector product\n",
    "\n",
    "Matrix and vector product is defined as follows:\n",
    "\n",
    "- $P \\in \\mathbb R^{mxn}$\n",
    "- $v \\in \\mathbb R^n$\n",
    "\n",
    "\n",
    "\n",
    "- $a = P v$,  Note that $n$ should be the same\n",
    "\n",
    "\n",
    "\n",
    "- $a = [P_1 \\cdot v, P_2 \\cdot v, P_m \\cdot v]$\n",
    "\n",
    "- $a \\in \\mathbb R^m$\n",
    "\n",
    "\n",
    "$P$ = $\\begin{bmatrix}\n",
    "  <- P_1 -> \\\\ \n",
    "  <- P_2 -> \\\\\n",
    "  ... \\\\\n",
    "  <- P_m ->\n",
    "\\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "65e5fea8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mat P:\n",
      " [[1 2 3]\n",
      " [4 5 6]]\n",
      "dim of mat P: (2, 3)\n",
      "mat v:\n",
      " [1 2 3]\n",
      "dim of mat v: (3,)\n",
      "=====\n",
      "mat c:\n",
      " [14 32]\n",
      "dim of mat c: (2,)\n"
     ]
    }
   ],
   "source": [
    "P = np.array([\n",
    " [1,2,3],\n",
    " [4,5,6]\n",
    "])\n",
    "\n",
    "v = np.array(\n",
    " [1,2,3])\n",
    "\n",
    "print(\"mat P:\\n {}\".format(P))\n",
    "print(\"dim of mat P: {}\".format(P.shape))\n",
    "print(\"mat v:\\n {}\".format(v))\n",
    "print(\"dim of mat v: {}\".format(v.shape))\n",
    "\n",
    "c=P@v # @ dot product, '*' in numpy is element-wise operation, and it might broadcast based on the dimensions\n",
    "print(\"=====\")\n",
    "print(\"mat c:\\n {}\".format(c))\n",
    "print(\"dim of mat c: {}\".format(c.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f659b008",
   "metadata": {},
   "source": [
    "### Matrix multiplications\n",
    "\n",
    "Matrix multiplications are dot product of rows and columns.\n",
    "\n",
    "- $P \\in \\mathbb R^{mxn}$\n",
    "- $Q \\in \\mathbb R^{nxk}$\n",
    "\n",
    "\n",
    "\n",
    "This picture describes matrix multiplications. Element $i$, $j$ of output matrix is the dot product of $i$th row of $a$ and $j$th column of $b$\n",
    "\n",
    "\n",
    "<div>\n",
    "<img src=\"figures/khan_academy_matrix_product.png\" width=\"350\"/>\n",
    "</div>\n",
    "\n",
    "([image source](https://www.khanacademy.org/math/precalculus/x9e81a4f98389efdf:matrices/x9e81a4f98389efdf:multiplying-matrices-by-matrices/a/multiplying-matrices))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b84f019",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3ebede4c",
   "metadata": {},
   "source": [
    "## Credits\n",
    "\n",
    "This notebook uses the contents from the followring materials:\n",
    "\n",
    "1. [Linear Algebra Review](https://stanford-cs221.github.io/autumn2021-extra/modules/prerequisites/linear-algebra.pdf)\n",
    "1. [Linear Algebra](https://ml-cheatsheet.readthedocs.io/en/latest/linear_algebra.html)\n",
    "\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b47b81b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('pytorch_env')",
   "language": "python",
   "name": "python3812jvsc74a57bd03fa8f9c45b60853bb3d615f814548b5337da45fc6ea40b493b018f80b808d764"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}