FIx ch06

Author: ironartisan

docs/ch06/ch06.md

@@ -191,7 +191,7 @@ Loss = \alpha \cdot \mathcal L_{CE} + \beta \cdot \tau ^ 2 \cdot \mathcal L_{KL}
 $$
 
 
-   一般情况下， 要保持 $\alpha + \beta = 1$。 实践中 $\alpha$ 通常取 $0.1$， 而 $\beta$ 通常取 $0.9$。至于为什么要对软损失部分的 $\operatorname {KL}$ 乘一个 $\tau ^ 2$， 简单的解释是为了保持软损失和硬损失在梯度上的平衡， 而具体的推导过程请选择性的阅读 [6.7](#67-损失函数部分推导选修) 节。
+   一般情况下， 要保持 $\alpha + \beta = 1$。 实践中 $\alpha$ 通常取 $0.1$， 而 $\beta$ 通常取 $0.9$。至于为什么要对软损失部分的 $\operatorname {KL}$ 乘一个 $\tau ^ 2$， 简单的解释是为了保持软损失和硬损失在梯度上的平衡， 而具体的推导过程请选择性的阅读 [损失函数部分推导](#损失函数部分推导选修)节。
 
 
 
@@ -847,6 +847,129 @@ print('best_Test_Acc = ', best_test_acc)
     best_Test_Acc =  95.89
 ```
 
+## 损失函数部分推导(选修)
+
+
+### Softmax函数求导
+
+
+  假设对于一个任意的 Logits 向量 $\mathbf z = [z_1, z_2, ..., z_{K}]\in \mathbb{R}^{1\times K}$, 其中 $K$ 是数据集的类别数。通过带温度的Softmax函数计算后得到向量 $\mathbf s = [s_1(\tau), s_2(\tau), ..., s_K(\tau)]$ ，其中 $s_i(\tau)$ 的定义为 : 
+$$
+s_i(\tau) = \frac{e^{z_i/ \tau}}{\sum_{j=1}^K e^{z_j / \tau}}
+$$
+
+
+  对于任意 $z_k \in \mathbf z$ ,   $s_i(\tau)$ 对 $z_k$ 的偏导分为两种情况:
+
+
+   当 $i = k$ 时， 有
+$$
+\begin{align*}
+  \frac{\partial s_i(\tau)}{\partial z_k}
+  &= \frac{\partial}{\partial z_k}\frac{e^{z_k / \tau}}{\sum_{j=1}^K e^{z_j / \tau}} \\ 
+  &= \frac{\frac{\partial}{\partial z_k}e^{z_k / \tau}\ \sum_{j=1}^K e^{z_j / \tau} - e^{z_k / \tau}\ \frac{\partial}{\partial z_k}\sum_{j=1}^K e^{z_j / \tau}}{\left( \sum_{j=1}^K e^{z_j / \tau}\right) ^ 2} \\
+  &= \frac{\frac{1}{\tau}\ e^{z_k/ \tau}}{\sum_{j=1}^K e^{z_j / \tau}} - \frac{e^{z_k/ \tau }\ \frac{1}{\tau}\ e^{z_k/ \tau }}{\left( \sum_{j=1}^K e^{z_j / \tau}\right) ^ 2}\\ 
+  &= \frac{1}{\tau} (s_k(\tau) - s_k(\tau)\ s_k(\tau)) \\
+  &= \frac{1}{\tau}\ s_k(\tau)\ (1 - s_k(\tau))
+\end{align*}
+$$
+
+   当 $i \neq k$ 时， 有
+$$
+\begin{align*}
+  \frac{\partial s_i(\tau)}{\partial z_k}
+  &= \frac{\partial}{\partial z_k}\frac{e^{z_i / \tau}}{\sum_{j=1}^K e^{z_j / \tau}} \\ 
+  &= \frac{\frac{\partial}{\partial z_k}e^{z_i / \tau}\ \sum_{j=1}^K e^{z_j / \tau} - e^{z_i / \tau}\ \frac{\partial}{\partial z_k}\sum_{j=1}^K e^{z_j / \tau}}{\left( \sum_{j=1}^K e^{z_j / \tau}\right) ^ 2} \\
+  &= 0 - \frac{e^{z_i/ \tau }\frac{1}{\tau}e^{z_k/ \tau }}{\left(\sum_{j=1}^K e^{z_j / \tau}\right) ^ 2}\\ 
+  &= -\frac{1}{\tau}\ s_i(\tau)\ s_k(\tau)
+\end{align*}
+$$
+
+   因此， 对于 $\varphi (z_i)$ 对 $z_k$ 偏导有
+$$
+\frac{\partial s_i(\tau)}{\partial z_k} =
+\left\{
+\begin{matrix}
+\frac{1}{\tau}\ s_k(\tau)\ (1 - s_k(\tau))& \text{if } i = k \\
+-\frac{1}{\tau}\ s_i(\tau)\ s_k(\tau) & \text{if } i \neq k
+\end{matrix}
+\right.
+$$
+
+
+### 硬损失CE求导
+
+  对于学生模型的输出logits $\mathbf{v} = [v_1, v_2, ..., v_K] \in \mathbb{R}^{1\times K}$ 中任意一个 $v_k$ ， 硬损失 $ \mathcal L_{CE}$ 对 $v_k$ 的梯度为 :
+$$
+\begin{align*}
+\mathcal L_{CE} &= \operatorname {CE}(\mathbf q(\tau = 1), \mathbf y) \\ 
+&= \sum _ {j=1}^K - y_j \log q_j(\tau = 1) \\
+\frac{\partial \mathcal L_{CE}}{\partial v_k} &= \frac{\partial}{\partial v_k}\sum _ {j=1}^K - y_j \log q_j(\tau = 1)\\
+&= \frac{\partial}{\partial v_k}\sum _ {j=1, j\neq k}^K - y_{j} \log q_j(\tau = 1) + \frac{\partial}{\partial v_k} - y_k \log q_k(\tau = 1)\\
+&= \sum _ {j=1, j\neq k}^K - y_{j} \frac{1}{q_j(\tau=1)}\ \frac{-1}{\tau=1}q_j(\tau=1)q_k(\tau=1) \\ &\ \ \ \ \ \ - y_k \frac{1}{q_k(\tau=1)}\frac{1}{\tau=1}q_k(\tau=1)(1-q_k(\tau=1))\\
+&= \frac{1}{1}(1-y_k)q_k(\tau=1) - \frac{1}{1}y_k(1-q_k(\tau=1))\\
+&= q_k(\tau=1) - y_k \\
+\end{align*}
+$$
+
+
+### 软损失KL求导
+
+  软损失$\mathcal L_{KL}$为教师模型的软标签$\mathbf p(\tau)$ 和学生模型的软标签 $\mathbf q(\tau)$  的KL散度。$\mathcal L_{KL}$ 对于学生模型的输出中任意一个 $v_k$ 的梯度为
+$$
+\begin{align*}
+\mathcal{L}_{\operatorname{KL}} &= \operatorname {KL}(\mathbf{q}(\tau), \mathbf{p}(\tau)) \\
+&= \sum_{j=1}^{K} \ p_j(\tau) \log \frac{p_j(\tau)}{q_j(\tau)} \\
+\frac{\partial \mathcal{L}_{\operatorname{KL}}}{\partial v_k} &= \frac{\partial}{\partial v_k} \sum_{j=1}^{K} \ p_j(\tau) \log \frac{p_j(\tau)}{q_j(\tau)} \\
+&= \frac{\partial}{\partial v_k} \sum_{j=1}^{K} \left( \ p_j(\tau) \log {p_j(\tau)} - p_j(\tau)\log{q_j(\tau)} \right) \\
+&= \frac{\partial}{\partial v_k} \left(\sum_{j=1}^{K} - p_j(\tau)\log{q_j(\tau)}\right) \\
+&= \frac{\partial}{\partial v_k} \left( \sum_{j=1, j\neq k} ^ K -p_j(\tau) \log q_j(\tau)  -p_k(\tau) \log q_k(\tau) \right) \\
+&= \sum_{j=1, j\neq k}^K \left( -p_j(\tau) \frac{\partial}{\partial v_k} \log q_j(\tau)\right) - \frac{\partial}{\partial v_k} p_k(\tau) \log q_k(\tau)\\
+&= \sum_{j=1, j\neq k}^K -\frac{p_j(\tau)}{q_j(\tau)}\left[ -\frac{1}{\tau}q_j(\tau)q_k(\tau) \right] - \frac{p_k(\tau)}{q_k(\tau)}\left[ \frac{1}{\tau} q_k(\tau)(1 - q_k(\tau))\right]\\ 
+&\approx \frac{1}{\tau}  \sum_{j=1, j\neq k}^K p_j(\tau) q_k(\tau) - \frac{1}{\tau} p_k(\tau)(1 - q_k(\tau)) \ \ \ \text{where} \sum_{j=1}^K p_j(\tau)\approx 1\\
+&= \frac{1}{\tau} (1 - p_k(\tau))q_k(\tau) - \frac{1}{\tau} p_k(\tau)(1 - q_k(\tau)) \\
+&= \frac{1}{\tau} \left[ q_k(\tau) - p_k(\tau)q_k(\tau) - p_k(\tau) + p_k(\tau)q_k(\tau)\right] \\
+&= \frac{q_k(\tau) - p_k(\tau)}{\tau}
+\end{align*}
+$$
+
+### 泰勒逼近
+
+  对于 $e^x$ ， 当 $x$ 趋于 0 的时候有 $e^x \approx 1 + x + ...$
+
+  最终， 硬损失CE和软损失KL对于$v_k$ 的梯度为:
+$$
+\left\{
+\begin{matrix}
+\frac{\partial \mathcal L_{CE}}{\partial v_k} = q_k(\tau=1) - y_k
+\\
+\frac{\partial \mathcal{L}_{\operatorname{KL}}}{\partial v_k} = \frac{1}{\tau}(q_k(\tau) - p_k(\tau))
+\end{matrix}
+\right.
+$$
+
+  对于$\frac{\partial \mathcal L_{CE}}{\partial v_k}$ 展开有 :
+$$
+\begin{align*}
+\frac{\partial \mathcal L_{CE}}{\partial v_k} &= q_k(\tau=1)-y_k\\
+&= \frac{e^{v_k}}{\sum_{j=1}^K e^{v_j}} - y_k \\
+&\approx \frac{1+v_k}{\sum_{j=1}^K 1 + v_j} - y_k  , \text{where} \sum v_j = 0\\
+&= \frac{1+v_k}{K} - y_k\\
+\end{align*}
+$$
+​
+  对于 $\frac{\partial \mathcal{L}_{\operatorname{KL}}}{\partial v_k}$ 展开有 :
+$$
+\begin{align*}
+\frac{\partial \mathcal{L}_{\operatorname{KL}}}{\partial v_k} &= \frac{1}{\tau}(q_k(\tau) - p_k(\tau))\\
+&= \frac{1}{\tau} (\frac{e^{v_k/\tau}}{\sum_{j=1}^K e^{v_j / \tau}} - \frac{e^{u_k/ \tau}}{\sum_{j=1}^K e^{u_k/\tau}}) \\
+&\approx \frac{1}{\tau}(\frac{1 + v_k/\tau}{\sum_{j=1}^K (1 + v_j/\tau)} - \frac{1+u_k/\tau}{\sum_{j=1}^K (1+u_j/ \tau)})\\
+&= \frac{1}{\tau}(\frac{v_k/\tau - u_k}{K}) \\
+&= \frac{1}{K \ \tau^2} v_k - \frac{u_k}{K\tau}
+\end{align*}
+$$
+
+  此时可以发现， 硬损失中梯度对于$v_k$ 的部分时软损失的梯度中对于$v_k$部分的 $\tau ^ 2$ 倍， 所以在最终计算损失函数Loss的时候， 需要给 $\mathcal{L}_{\operatorname{KL}}$ 乘上一个 $\tau ^ 2$ 以平衡两个损失之间的梯度。
 
 
 ## 引用资料