4-process.tex

\section{Process}
\subsection{Overall Roadmap}

In order to obtain a smaller model, less training and better performance,
    we decided to make several improvements of models and training based on DCGAN\upcite{dcgan}.

Figure \ref{roadmap} is the overall technical roadmap of this study.

\begin{figure}
    \begin{center}
    \includegraphics[width=\textwidth]{figures/roadmap.pdf}
    \caption{Overall technical Roadmap}
    \label{roadmap}
    \end{center}
\end{figure}


\subsection{Research Process}

In previous model adjustments, we conducted several comparative tests to select suitable hyperparameters.

Figure \ref{norm_bach} and Figure \ref{norm_instance} show the results of model after 2 training epochs(about 5500 iterations per epoch) using batch normalization and instance normalization in that order.
It can be seen that the image quality improves under the same training amount after using instance normalization,
    which means that the convergence speed of the model is faster.

\begin{figure}
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/result_norm_batch.png}
        \caption{Test result using Batch Normalization after training 2 epochs}
        \label{norm_bach}
    \end{minipage}
        \hfill
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/result_norm_instance.png}
        \caption{Test result using Instance Normalization after training 2 epochs}
        \label{norm_instance}
    \end{minipage}
\end{figure}

In addition, we tested the effects of convolution kernel size and numbers of convolution filters on model performance and model size.
Figure \ref{conv_filter_16}, Figure \ref{conv_filter_24}, and Figure \ref{conv_filter_32} are test result that convolution filters using 16 times, 24 times, and 32 times,
    respectively, they are completed after training 15 epochs.
The times of convolution filter refers to the start amount of a convolution kernel.
It will be doubled in each next layer of the decoder and the encoder.
It can be seen that after the convolution filter number is lowered to 24x from 32x,
    the image quality does not decrease markedly under the same training amount.
But when the number decreases to 16x, the model dons't performs as well as 24x and 32x.
In order to keep the balacne between use-cost and performance,
    we finally chose a 24 times convolution filters.

\begin{figure}
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/result_conv_filter_16.png}
        \caption{Test result using 16x convolution filter after training 15 epochs}
        \label{conv_filter_16}
    \end{minipage}
        \hfill
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/result_conv_filter_24.png}
        \caption{Test result using 24x convolution filter after training 15 epochs}
        \label{conv_filter_24}
    \end{minipage}
    \begin{minipage}[t]{\linewidth}
        \centering
        \includegraphics[width=0.48\textwidth]{figures/result_conv_filter_32.png}
        \caption{Test result using 32x convolution filter after training 15 epochs}
        \label{conv_filter_32}
    \end{minipage}
\end{figure}

Figure \ref{conv_kernel_5} and \ref{conv_kernel_3} show the results after training 4 epochs of the 5×5 and 3×3 (transposed) convolution kernel,
    respectively, under a 24 times convolution filter.
It can be seen that switching from a 5×5 convolution kernel to a 3×3 size causes the image quality to degrade within an acceptable range,
    so we choose a 3×3 convolution kernel.

\begin{figure}
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/result_conv_kernel_5.png}
        \caption{Test result using 5×5 convolution kernel after training 4 epochs}
        \label{conv_kernel_5}
    \end{minipage}
        \hfill
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/result_conv_kernel_3.png}
        \caption{Test result using 3×3 convolution kernel after training 4 epochs}
        \label{conv_kernel_3}
    \end{minipage}
\end{figure}
Otherwise, we have done loads of experimental comparison about parameter initializer,
    learning rate, noise dimension, noise distribution, activation function,
    loss function, network layers and training method to get a better model structure,
    sets of hyperparameters and training method.

\subsection{Model Overview}

After the comparisons, we finally obtained the conditional facial image generation and adjustment model.
It is as shown in Figure \ref{smliegan}.

\begin{figure}
    \begin{center}
    \includegraphics[width=\textwidth]{figures/architecture.pdf}
    \caption{The Model of LittleGAN}
    \label{smliegan}
    \end{center}
\end{figure}

LittleGAN consists of 2 main components and 3 networks.
The main components are the encoder and decoder, used in 3 networks.
The networks are the generator, discriminator and adjustor.
In LittleGAN, the generator generates facial images from noise and facial attributes,
    and then the adjustor combines it with new attributes to obtain new images.


\subsection{Encoder and Decoder}


Main components, encoder and decoder, are used for the conversion of feature map and images.

The encoder can extract features from the image from global to local.
% local/global image feature please reference Baidu wiki
Using convolution, starting from an image of 128×128×3 (height × width × channels),
    the convolution with stride is continuously performed.
After each convolution, normalization, activation and dropout\upcite{dropout} are used.

The convolution layer uses a 5×5 convolution kernel.
Convolution filter number is 24 in the first layer, doubled in each next layer.
The image dimension is reduced as the features are extracted by using the convolution with stride,
    leading to fewer computational load for each successive layer.

Instance Normalization\upcite{instance} is used as the normalization layer.
To preserve the image for more information, Leaky ReLU\upcite{leaky} is used as the activation function.
In order to prevent over-fitting,
    dropout\upcite{dropout} is used during each training batch to randomly select some neurons from each batch by to be hidden,
    thereby reducing the interaction between filters.

After the encoder outputs a feature map, it is read by the decoder to generate an image.
Using transposed convolution, the decoder uses the same number of filters,
    strides and convolution kernels as the encoder.
Also the same, normalization and activation are performed after each transposed convolution.

\subsection{Generator and Discriminator}

Following the GAN network paradigm, LittleGAN also trains using a discriminator and a generator.
The generator consists of a fully connected layer and the decoder.
Facial attributes are combined with noise as input to generate images by the generator.
The discriminator consists of the encoder and 2 fully connected layers.
It uses images as input then extracts attributes and the probability of image comes from data.


In the generator, through a fully connected layer, the input are transformed into feature map(in 8×8×256 size).
Then the image is generated from local to global by the decoder's layers.
Finally a 3 filters convolution without stride is performed and used.
The tanh activation function is used to obtain a three-channel (RGB) color image of 128×128×3 size.
Figure \ref{net_generator} is the structure graph of generate network.

\begin{figure}
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/net_generator.pdf}
        \caption{Generator Network Structure of LittleGAN}
        \label{net_generator}
    \end{minipage}
        \hfill
    \begin{minipage}[t]{0.48\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/net_discriminator.pdf}
        \caption{Discriminator Network Structure of LittleGAN}
        \label{net_discriminator}
    \end{minipage}
\end{figure}
The discriminator uses the encoder to extract the feature map from the input image first. The feature map then goes through a fully connected layer to the result.
Then on one hand,  the result turns to the probability indicating that the image comes from real data. It uses the Sigmoid activation function to normalize the result to [0,1],
On the other hand, the result turns to attributes, also normalized to [0,1] using the Sigmoid.
Figure \ref{net_discriminator} is the structure graph of discriminator network.

As for the discriminator's loss function, since the Sigmoid is used to activate the output,
    in order to ensure that the network parameters will not dead,
    we expect the loss of the discriminator output to [0.02,0.98] instead of [0,1].
Since discriminator not only outputs probability of image comes from data but also contains the facial attribute information implied by the image,
    its loss includes the GAN loss and the conditional judgment loss.
In the discriminator, we expect the output from real image close to 0.98, while the output from fake image close to 0.02.
Similarly, we expect the facial attribute information output by the discriminator is as consistent as possible with the real data.
Equation \eqref{loss_d_gan}, Equation \eqref{loss_d_c}, and Equation \eqref{loss_d} are the GAN loss,
    conditional adjustment loss and total loss of the discriminator, respectively.

% Todo: Add comment for equation
\begin{equation}
    \mathcal{L}_{D-GAN}(D,G)=
    \mathbb{E}_{y\thicksim P_{data}}[(D(y)-0.98)^2]+
    \mathbb{E}_{c\thicksim P_{data},z\thicksim P_{fake}}[(D(G(c,z))-0.02)^2]
    \label{loss_d_gan}
\end{equation}


\begin{equation}
    \mathcal{L}_{C}(C)=
    \mathbb{E}_{c,y\thicksim P_{data}}[(c-C(y))^2]
    \label{loss_d_c}
\end{equation}

\begin{equation}
    \mathcal{L}_{D}(C,D,G)=
    \mathcal{L}_{D-GAN}(D,G)+
    \mathcal{L}_{C}(C)
    \label{loss_d}
\end{equation}

We add a mean absolute error to the generator's loss function.
We expect overall color and layout to be more realistic while keeping the diversity generation results in this way.
The two goals of the generator is to generate an image that tricks the discriminator into labeling the forgery as real data,
    and to have the discriminator output an attribute close to generation condition.
Therefore, the loss function of the generator includes into the discriminator's loss and the mean absolute error loss.
Equation \eqref{loss_g_gan}, Equation \eqref{loss_g_l1}, and Equation \eqref{loss_g} are the discriminator loss,
    mean absolute error loss and total loss of the generator, respectively.
The $\lambda$ in the total loss in the experiment is 0.02.

\begin{equation}
    \mathcal{L}_{G-GAN}(C,D,G)=
    \mathbb{E}_{c\thicksim P_{data},z\thicksim P_{fake}}[(0.98-D(G(c,z)))^2]+
    \mathbb{E}_{c\thicksim P_{data},z\thicksim P_{fake}}[(c-C(G(c,z)))^2]
    \label{loss_g_gan}
\end{equation}

\begin{equation}
    \mathcal{L}_{G-L1}(G)=
    \mathbb{E}_{c,y\thicksim P_{data},z\thicksim P_{fake}}[|y-G(c,z)|]
    \label{loss_g_l1}
\end{equation}

\begin{equation}
    \mathcal{L}_{G}(C,D,G)=
    \mathcal{L}_{G-GAN}(C,D,G)+
    \lambda \mathcal{L}_{G-L1}(G)
    \label{loss_g}
\end{equation}

Our goal is to minimize the loss function of the generator and the discriminator,
    so the optimization goal of the generator against the network is Equation \eqref{gan_target}

\begin{equation}
    \arg \min_{C,D} \max_{G}V(C,D,G)=\mathcal{L}_{D}(C,D,G)+\mathcal{L}_{G}(C,D,G)
    \label{gan_target}
\end{equation}


\subsection{Adjustor Network}
The adjustor consists of the decoder and the encoder from the GAN as well as several shortcut channels and a combined channel of adjustment conditions.
It uses the encoder to extract both global and local features.
Each layer’s features are relayed to the equivalent layer of the decoder network through a shortcut channel.
This allows each layer of the network to focus on data compression rather than the preservation of all the details found in the original image.
Adjustor uses the decoder to combine the extracted features and adjustment conditions step by step to generate an image from detail to global.

Figure \ref{net_adjustor} is the structure graph of adjustor network.

\begin{figure}
    \begin{center}
    \includegraphics[width=0.48\textwidth]{figures/net_adjustor.pdf}
    \caption{Adjustor Network Structure of LittleGAN}
    \label{net_adjustor}
    \end{center}
\end{figure}

In the adjustor network, the decoder and the encoder parameters are shared with GAN's.
We use GAN loss and absolute mean error as a loss function for the adjustor network.
Equation \eqref{loss_a_gan}, Equation \eqref{loss_a_l1}, and Equation \eqref{loss_u} are the GAN loss,
    mean absolute error loss and total loss of the adjustor, respectively.

\begin{equation}
    \mathcal{L}_{A-GAN}(C,D,A)=
    \mathbb{E}_{c,y\thicksim P_{data}}[(0.98-D(A(c,y)))^2]+
    \mathbb{E}_{c,y\thicksim P_{data}}[(c-C(A(c,z)))^2]
    \label{loss_a_gan}
\end{equation}

\begin{equation}
    \mathcal{L}_{A-L1}(A)=
    \mathbb{E}_{c,y\thicksim P_{data},z\thicksim P_{fake}}[|y-A(c,z)|]
    \label{loss_a_l1}
\end{equation}

\begin{equation}
    \mathcal{L}_{A}(C,D,A)=
    \mathcal{L}_{A-GAN}(C,D,A)+
    \lambda \mathcal{L}_{A-L1}(A)
    \label{loss_u}
\end{equation}

\subsection{Model Training}
There are two main improvements to the training method of the model:
    We apply the gradient penalty\upcite{wgan-gp} to the discriminator, improving the convergence speed and the stability;
    We use the partition training that proposed by us,
    enabling the network to adjust the previous network layer more effectively
    so that speed up the convergence.

The specific operation of the partition training proposed by us is as follows.
The network layer parameters that need to be trained in the model are grouped according to the function and the number of parameters.
When training each group of parameters,
    other parameters are fixed and only the parameters in the group are back-propagated.