study_auto_encoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Belief Propagation Auto-Encoder - Base Study\n",
    "This notebook provides a MWE of the NBP AE with the example of a (31,16) code.\n",
    "For accessing the results of the paper please refer to the corresponding branches of the repository (one branch per study presented in the paper)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "#-*- coding: utf-8 -*-\n",
    "\n",
    "\n",
    "# sanity check\n",
    "from platform import python_version\n",
    "print(f'python: {python_version()}')\n",
    "\n",
    "import tensorflow as tf\n",
    "print(f'tensorflow: {tf.version.VERSION}')\n",
    "\n",
    "physical_devices_available = tf.config.list_physical_devices()\n",
    "print(f'Available physical devices: {physical_devices_available}')\n",
    "\n",
    "import os\n",
    "os.environ['TF_GPU_THREAD_MODE']='gpu_private'\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Size & Reference Code (Optional)\n",
    "This cell set the code size that will be studied using the argument '(n,k)=(31,16)'.\n",
    "In this example a reference BCH (31,16) code is used for comparison with the code learned by the AE.\n",
    "If one does not want to compare the code learned by the AE with such a reference code, the 'use_reference_code' should be set to 'False'.\n",
    "Different code can be used for reference, although they should have the same size and rate to that of the AE configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Study code size\n",
    "(n,k)=(31,16)\n",
    "\n",
    "# Get the reference code G and H matrices\n",
    "use_reference_code = True\n",
    "if use_reference_code:\n",
    "    codename = f\"BCH_{n}_{k}\" # name must match that of a npz file at the \"encoders/linearblockencoders_reference/\" path\n",
    "    code_path = os.path.join(\"./\",\"encoders\",\"linearblockencoders_reference\", f\"{codename}.npz\")\n",
    "\n",
    "    code_file = np.load(code_path)\n",
    "    # print(code_file.files)\n",
    "\n",
    "    G_sys = tf.convert_to_tensor(code_file['G'], dtype=tf.float32)\n",
    "    H_sys = tf.convert_to_tensor(code_file['H_systematic'], dtype=tf.float32)\n",
    "    H_nsys = tf.convert_to_tensor(code_file['H_non_systematic'], dtype=tf.float32)\n",
    "\n",
    "    print(\"Reference Code Matrices:\")\n",
    "    print(\"Standard Form Generator Matrix:\")\n",
    "    tf.print(G_sys,summarize=-1)\n",
    "    print(\"Standard Form Parity-Check Matrix:\")\n",
    "    tf.print(H_sys,summarize=-1)\n",
    "    print(\"Standard Form Parity-Check Matrix:\")\n",
    "    tf.print(H_nsys,summarize=-1)\n",
    "    \n",
    "    print(\"Check that G.H equals to 0:\")\n",
    "    tf.print(tf.matmul(G_sys,tf.transpose(H_sys))%2,summarize=-1)\n",
    "    tf.print(tf.matmul(G_sys,tf.transpose(H_nsys))%2,summarize=-1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datasets Generation\n",
    "The following cell define the training, validation and test datasets generation mechanisms including number of training epochs, steps per epochs, batch size, etc.\n",
    "While validation and test data are randomly sampled from all possible information words, the training data is sample from the basis, all-zero and all-one vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataset import random_messages_dataset, random_messages_base_all_zero_all_one_dataset\n",
    "\"\"\"\n",
    "#Dataset logic:\n",
    ".take() allow to generate a fixed number of batch of the dataset\n",
    ".cache() allow to use cached data (i.e doesn't execute previous dataset generation instructions)\n",
    ".shuffle() suffle elements inside batches by picking randomly elmts in a buffer of size buffer_size. similarly to batching procedure, shuffling method of model.fit isn't called when using dataset. it's important to manualy specify a shuffling strategy then.\n",
    ".prefetch() is used to overlap the processing time of the data producer and data consumer. should be used as last step. \n",
    "Here the step before .cache() should be done once and for all at first call of the train_dataset object. The operations after .cache() should be called at each time the dataset is called (ie for each batch in the training).\n",
    "All the steps before .prefetch() are supposed to be prepared during the training step of the data consumer to be ready when the consumer needs more data.\n",
    "\"\"\"\n",
    "\n",
    "#TRAIN DATASET\n",
    "epochs=100\n",
    "steps_per_epoch=25\n",
    "train_batch_size = 64\n",
    "train_seed = 42\n",
    "train_dataset = random_messages_base_all_zero_all_one_dataset(\n",
    "        k, \n",
    "        batch=train_batch_size, \n",
    "        prefetch=tf.data.AUTOTUNE, \n",
    "        seed=train_seed\n",
    "    ).take(steps_per_epoch * epochs).cache()\n",
    "\n",
    "#VALIDATION DATASET\n",
    "validation_seed = 43\n",
    "val_batch_size = 64\n",
    "validation_dataset = random_messages_dataset(\n",
    "        k, \n",
    "        batch=val_batch_size, \n",
    "        prefetch=tf.data.AUTOTUNE, \n",
    "        seed=validation_seed\n",
    "    ).take(int(8000 / val_batch_size)).cache()\n",
    "\n",
    "#TEST DATASET\n",
    "test_batch_size = 100  \n",
    "test_seed = 44\n",
    "test_dataset = random_messages_dataset(\n",
    "        k,\n",
    "        batch=test_batch_size,  \n",
    "        prefetch=tf.data.AUTOTUNE,\n",
    "        seed=test_seed,  \n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Noise Settings\n",
    "The following cell, defines the training and evaluation noise levels ($E_b/N_0$) and compute the corresponding SNR levels based on the code rate.\n",
    "The training noise level is set to $E_b/N_0 = 4 \\mathrm{dB}$ while the model is eevaluated after training between $0$ and $7$ dB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tools import ebno_db_to_snr_db\n",
    "\n",
    "#Training/eval noise level settings\n",
    "ebn0_training_dbs = 4\n",
    "ebn0_eval_dbs = tf.range(0, 7, delta=1, dtype=tf.float32)\n",
    "noise_power_training_dbs = -ebno_db_to_snr_db(ebn0_training_dbs, k/n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Study Folders\n",
    "The following cells creates all the folders necessary to storing the results of the training (BER,BLER,Bit Error Count (BEC) aswell as the corresponding +/-5% confidence intervals, the models parameters and the Tensorboard data visualisation).\n",
    "\n",
    "Note: Depending on the folder location and model name, the path might exceed the MAX_PATH length on Windows system.\n",
    "If an utf8 encoding error is observed during the execution of the model, one might need to enable long path in the file system configuration:\n",
    "- Run Regedit\n",
    "- navigate to Computer\\HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\FileSystem (you can past this into the address bar in Regedit)\n",
    "- set LongPathsEnabled to 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tools import create_paths_and_summaries\n",
    "\n",
    "path = path = os.path.join(\"study-ae-{n}-{k}\".format(n=n,k=k))\n",
    "paths_and_summaries = create_paths_and_summaries(path, 'Eb/N0 (dB)', ebn0_eval_dbs)\n",
    "summary_ber = paths_and_summaries.summary_ber\n",
    "summary_bler = paths_and_summaries.summary_bler\n",
    "summary_bec = paths_and_summaries.summary_bec\n",
    "summary_blec = paths_and_summaries.summary_blec\n",
    "summary_bpci_ber = paths_and_summaries.summary_bpci_ber\n",
    "summary_bpci_bler = paths_and_summaries.summary_bpci_bler\n",
    "models_path = paths_and_summaries.models_path\n",
    "tensorboard_path = paths_and_summaries.tensorboard_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define AE Model Configuration\n",
    "This cell defines the model creation routine that will be called at each model creation of the study protocol. The function should thus expose the various parameters usefull of the study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model definition\n",
    "from autoencoders import AutoEncoder\n",
    "from metrics import BitErrorRate, BlockErrorRate, BinomialProportionConfidenceInterval, BitErrorCount, BlockErrorCount\n",
    "\n",
    "noise_power_training_dbs = -ebno_db_to_snr_db(ebn0_training_dbs, k/n)\n",
    "\n",
    "def create_model(\n",
    "    n,                              \n",
    "    k,                             \n",
    "    build_dataset,                  \n",
    "    conf=\"A\",                       \n",
    "    learning_rate=1e-1,             \n",
    "    model_index=None,               \n",
    "    training_noise_power_db=0.0,  \n",
    "    train_model=True,\n",
    "    G=None,\n",
    "    H=None,\n",
    "    trainable_code=True,\n",
    "    trainable_decoder=True,\n",
    "    additional_confs=[],\n",
    "    name=None,\n",
    "):\n",
    "    \"\"\"\n",
    "    Model Creation Function\n",
    "\n",
    "    Args:\n",
    "        n (int): Code-words size\n",
    "        k (int): Information block size\n",
    "        build_dataset (dataset): A dataset with the characteristics of the training/validation dataset must be provided to build the model graph upon creation.\n",
    "        conf (str, optional) [default=\"A\"]: Type of decoders to be used (see 'decoders/decoder.py' for the different decoder avalaible). Configuration 'A' is the configuration of the decoder as described in the paper. \n",
    "        learning_rate (float, optional) [default=1e-1]: Training LR.\n",
    "        model_index (int, optional) [default=None]: Model index appended to the end of auto-name generation (if name is None).\n",
    "        training_noise_power_db (float, optional) [default=0.0]: Noise power (in dB) used during training.\n",
    "        train_model (bool, optional) [default=True]: Wheter to train the model or not.\n",
    "        G ((k,n) tf.float32 tensor,optional) [default=None]: Generator matrix used for initialisation.\n",
    "        H ((n-k,n) tf.float32 tensor,optional) [default=None]: Parity-check matrix used for initialisation.\n",
    "        trainable_code (bool, optional) [default=True]: Whether to train the code of the AE or not (valid if train_model=True).\n",
    "        trainable_decoder (bool, optional) [default=True]: Whether to train the decoders weights of the AE or not (valid if train_model=True).\n",
    "        additional_confs (list, optional) [default=[]]: Additional configuration to be tested after the traing of the model defined as a list of parameters that should match that of the create_model() function.\n",
    "        name (str, optional) [default=None]: Name of the model (that will among other things be used for data storage). If set to None, the model name will be defined following the automatic naming rule defined below.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Model's name definition\n",
    "    if name is None:\n",
    "        conf_string = str(conf) if conf is not None else \"\"\n",
    "        lr_string = str(learning_rate) if learning_rate is not None else \"\"\n",
    "        index_string = str(model_index) if model_index is not None else \"\"\n",
    "        train_model_string = str(train_model) if train_model is not None else \"\"\n",
    "        trainable_code_string = (\n",
    "            str(trainable_code) if trainable_code is not None else \"\"\n",
    "        )\n",
    "        name = f\"AE-{n}-{k}-{conf_string}-{index_string}\"\n",
    "    else:\n",
    "        name = name\n",
    "\n",
    "    # Auto-Encoder Creation with a fixed number of decoding iteration n_iter=5 \n",
    "    n_iter = 5\n",
    "    model = AutoEncoder(\n",
    "        n,\n",
    "        k,\n",
    "        n_iter,\n",
    "        conf,\n",
    "        training_noise_power_db=training_noise_power_db,\n",
    "        G=G,\n",
    "        H=H,\n",
    "        trainable_code=trainable_code,\n",
    "        trainable_decoder=trainable_decoder,\n",
    "        name=name,\n",
    "    )\n",
    "\n",
    "    # List of training and validation metrics\n",
    "    metric_list = [\n",
    "            BitErrorRate(name=\"BER\",from_logits=False),\n",
    "            BlockErrorRate(name=\"BLER\",from_logits=False),\n",
    "            BitErrorCount(name=\"BEC\",from_logits=False, mode=\"sum\"), \n",
    "            BlockErrorCount(name=\"BLEC\",from_logits=False, mode=\"sum\"), \n",
    "            BinomialProportionConfidenceInterval(monitor_class=BitErrorRate, monitor_params={\"name\":\"bpci_ber\",\"from_logits\":False},fraction=0.95, name=\"BPCI_BER\"),\n",
    "            BinomialProportionConfidenceInterval(monitor_class=BlockErrorRate, monitor_params={\"name\":\"bpci_bler\",\"from_logits\":False},fraction=0.95, name=\"BPCI_BLER\")\n",
    "        ]\n",
    "    \n",
    "    # Model compilation with RMSprop optimizer and BCE loss function.\n",
    "    model.compile(\n",
    "        optimizer=tf.keras.optimizers.RMSprop(learning_rate),\n",
    "        loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
    "        metrics=metric_list,\n",
    "    )\n",
    "\n",
    "    \n",
    "    # Build model graph using build dataset (one sample)\n",
    "    build_datum = list(build_dataset.take(1)) \n",
    "    model(build_datum[0][0])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Training Mechanisms\n",
    "The train_model function defines the study path where to store results and trained model configuration aswell as the training call-backs to be used during training. Finally it execute the model.fit function and reload the weights of the best model after the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from callbacks.defaults import default_training_callbacks, default_configuration_early_stopping, default_configuration_tensorboard, default_configuration_reduce_lr_on_plateau, default_configuration_model_checkpoint\n",
    "\n",
    "def train_model(\n",
    "    model,\n",
    "    models_path,\n",
    "    train_dataset,\n",
    "    validation_dataset,\n",
    "    tensorboard_path,\n",
    "    epochs=1000,\n",
    "    steps_per_epoch=25,\n",
    "):\n",
    "    \"\"\"\n",
    "    Model Training Function\n",
    "\n",
    "    Args:\n",
    "        model (tf model): Model to be trained (must be build and compiled).\n",
    "        models_path (str): Where to store the model training checkpoints.\n",
    "        train_dataset (tf dataset): The training dataset to be used to train the model.\n",
    "        validation_dataset (tf dataset): The validation dataset to be used to monitor the model progress during training.\n",
    "        tensorboard_path (str): The path for Tensorboard checkpoint (visualistion tool from Tensorflow).\n",
    "        epochs (int, optional) [default=1000]: Number of training epochs. This correspond to a maximum number as an early stopping call-back is used.\n",
    "        steps_per_epochs (int, optional) [default=25]: Number of steps per epochs.\n",
    "    \"\"\"\n",
    "\n",
    "    # Define the training call-backs\n",
    "    ckpt_path = os.path.join(models_path, \"checkpoint\",\"checkpoint.tf\")\n",
    "    #configuration_tensorboard = default_configuration_tensorboard(tensorboard_path)\n",
    "    configuration_earlystopping = default_configuration_early_stopping(\n",
    "        monitor=\"loss\", patience=100\n",
    "    )\n",
    "    configuration_reduce_lr_on_plateau = default_configuration_reduce_lr_on_plateau(\n",
    "        monitor=\"val_loss\", factor=0.8, patience=25\n",
    "    )\n",
    "    print(ckpt_path)\n",
    "    configuration_model_checkpoint = default_configuration_model_checkpoint(\n",
    "        filepath=ckpt_path\n",
    "    )     \n",
    "\n",
    "    callbacks = default_training_callbacks(\n",
    "        configuration_earlystopping=configuration_earlystopping,\n",
    "        #configuration_tensorboard=configuration_tensorboard,\n",
    "        configuration_reduce_lr_on_plateau=configuration_reduce_lr_on_plateau,\n",
    "        configuration_model_checkpoint=configuration_model_checkpoint,\n",
    "    )\n",
    "\n",
    "    # Start the model's training\n",
    "    with tf.device(\"/GPU:0\"):\n",
    "        model.fit(\n",
    "            x=train_dataset,  \n",
    "            validation_data=validation_dataset,\n",
    "            epochs=epochs,\n",
    "            steps_per_epoch=steps_per_epoch,\n",
    "            callbacks=callbacks,\n",
    "            verbose=1,\n",
    "        )\n",
    "        \n",
    "    # Reload best weights at the end of the training\n",
    "    model.load_weights(ckpt_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Evaluation Mechanisms\n",
    "The evaluate_model function defines the evaluation process of the model.\n",
    "The ci_condition methods define the evaluation stopping criterion based on confidence intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from callbacks import BatchTerminationCallback\n",
    "\n",
    "\n",
    "def ber_ci_condition(_, logs):\n",
    "    # Return true when the BER confidence interval is smaller than 10% of the estimated BER value\n",
    "    if 'BPCI_BER' in logs:\n",
    "        epsilon = 1e-7\n",
    "        (ci_span, ci_low, ber, ci_high) = logs['BPCI_BER']\n",
    "        return (ci_span)/(ber+epsilon) < 0.1\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "def bler_ci_condition(_, logs):\n",
    "    # Return true when the BLER confidence interval is smaller than 10% of the estimated BER value\n",
    "    if 'BPCI_BLER' in logs:\n",
    "        epsilon = 1e-7\n",
    "        (ci_span, ci_low, bler, ci_high) = logs['BPCI_BLER']\n",
    "        return (ci_span)/(bler+epsilon) < 0.1\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "def evaluate_model(\n",
    "        summary_ber, \n",
    "        summary_bler, \n",
    "        summary_bec, \n",
    "        summary_blec, \n",
    "        summary_bpci_ber, \n",
    "        summary_bpci_bler, \n",
    "        models_path, \n",
    "        model, \n",
    "        ebn0_eval_dbs, \n",
    "        test_dataset, \n",
    "        k, \n",
    "        n\n",
    "    ):\n",
    "    \"\"\"\n",
    "    Model Evaluation Function\n",
    "\n",
    "    Args:\n",
    "        summary_ber (Summary): Summary file where to store the BER evaluation metric.\n",
    "        summary_bler (Summary): Summary file where to store the BLER evaluation metric. \n",
    "        summary_bec (Summary): Summary file where to store the (Bit Error Count) BEC evaluation metric. \n",
    "        summary_blec (Summary): Summary file where to store the (Block Error Count) BLEC evaluation metric. \n",
    "        summary_bpci_ber (Summary): Summary file where to store the (Binomial Proportion Confidence Interval on the BER) BPCI_BER evaluation metric. \n",
    "        summary_bpci_bler (Summary): Summary file where to store the (Binomial Proportion Confidence Interval on the BLER) BPCI_BLER evaluation metric. \n",
    "        models_path (str): Path where to store the model. \n",
    "        model (tf model): Model to be evaluated. \n",
    "        ebn0_eval_dbs ([float]): List of Eb/No level to be used for model evaluation. \n",
    "        test_dataset (tf dataset): Dataset to be used for evaluation\n",
    "        k (int): Information block size.\n",
    "        n (int): Code block size.\n",
    "    \"\"\"\n",
    "    # Initiate the metric list\n",
    "    bers = []\n",
    "    blers = []\n",
    "    becs = []  \n",
    "    blecs = []\n",
    "    bpci_bers = [] \n",
    "    bpci_blers = []\n",
    "    \n",
    "    # mean(Es) assumed to be 1\n",
    "    noise_power_eval_dbs = -ebno_db_to_snr_db(ebn0_eval_dbs, k / n)\n",
    "    snr_eval_dbs = ebno_db_to_snr_db(ebn0_eval_dbs, k / n)\n",
    "\n",
    "    # add SNR values\n",
    "    summary_ber[\"SNR(dB)\"] = snr_eval_dbs.numpy()\n",
    "    summary_bler[\"SNR(dB)\"] = snr_eval_dbs.numpy()\n",
    "    summary_bec[\"SNR(dB)\"] = snr_eval_dbs.numpy()\n",
    "    summary_blec[\"SNR(dB)\"] = snr_eval_dbs.numpy()\n",
    "    summary_bpci_ber[\"SNR(dB)\"] = snr_eval_dbs.numpy()\n",
    "    summary_bpci_bler[\"SNR(dB)\"] = snr_eval_dbs.numpy()\n",
    "\n",
    "    # Evaluate model for each eval Eb/No levels\n",
    "    for ebn0_eval_db, noise_power_eval_db in zip(ebn0_eval_dbs, noise_power_eval_dbs):\n",
    "        print(\n",
    "            f\"evaluating {model.name} at Eb/N0 [dB]: {ebn0_eval_db} / N0 [dB]: {noise_power_eval_db}\"\n",
    "        )\n",
    "        model.channel.noise_power_db=noise_power_eval_db\n",
    "\n",
    "        termination_callback = BatchTerminationCallback(ber_ci_condition)\n",
    "        summary = model.evaluate(\n",
    "            test_dataset,  # validation_dataset,\n",
    "            steps=25000,\n",
    "            return_dict=True,\n",
    "            callbacks=[termination_callback]\n",
    "        )\n",
    "        \n",
    "        (ber_ci_span, ci_min, ber_bpci_metric, ci_max) = summary['BPCI_BER']\n",
    "        (bler_ci_span, ci_min, bler_bpci_metric, ci_max) = summary['BPCI_BLER']\n",
    "        \n",
    "        #print(summary)\n",
    "        ber = summary[\"BER\"]  \n",
    "        bler = summary[\"BLER\"] \n",
    "        bec = summary[\"BEC\"]  \n",
    "        blec = summary[\"BLEC\"] \n",
    "        bpci_ber = summary[\"BPCI_BER\"]  \n",
    "        bpci_bler = summary[\"BPCI_BLER\"] \n",
    "        bers.append(ber) \n",
    "        blers.append(bler)\n",
    "        becs.append(bec) \n",
    "        blecs.append(blec)\n",
    "        bpci_bers.append(bpci_ber) \n",
    "        bpci_blers.append(bpci_bler)\n",
    "        \n",
    "        print(f\"Eb/N0: {ebn0_eval_db} BER: {ber} 95% CI: {ber_ci_span} - BEC: {bec} - BLER: {ber} 95% CI: {bler_ci_span} - BLEC: {blec}\")\n",
    "\n",
    "    # learned_code\n",
    "    G, H = model.code_generator(None)\n",
    "\n",
    "    # Store results and model\n",
    "    summary_ber[model.name] = bers\n",
    "    summary_bler[model.name] = blers\n",
    "    summary_bec[model.name] = becs\n",
    "    summary_blec[model.name] = blecs\n",
    "    summary_bpci_ber[model.name] = bpci_bers\n",
    "    summary_bpci_bler[model.name] = bpci_blers\n",
    "\n",
    "    model_path = os.path.join(models_path, model.name)\n",
    "    encoder_path = os.path.join(model_path, model.encoder.name)\n",
    "    decoder_path = os.path.join(model_path, model.decoder.name)\n",
    "    code_generator_path = os.path.join(model_path, model.code_generator.name)\n",
    "    matrices_path = os.path.join(model_path, \"matrices\")\n",
    "\n",
    "    print(model_path, encoder_path, decoder_path, code_generator_path)\n",
    "    os.makedirs(encoder_path, exist_ok=True)\n",
    "    os.makedirs(decoder_path, exist_ok=True)\n",
    "    os.makedirs(code_generator_path, exist_ok=True)\n",
    "    os.makedirs(matrices_path, exist_ok=True)\n",
    "    print(f\" saving model {model.name} in {model_path}\")\n",
    "\n",
    "    model.encoder.save(encoder_path, overwrite=True)\n",
    "    model.decoder.save(decoder_path, overwrite=True)\n",
    "    model.code_generator.save(code_generator_path, overwrite=True)\n",
    "    np.savetxt(os.path.join(matrices_path,\"G.csv\"), np.array(G), fmt=\"%i\")\n",
    "    np.savetxt(os.path.join(matrices_path,\"H.csv\"), np.array(H), fmt=\"%i\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Study Protocol\n",
    "The following cell define the protocol of the study. What are the configuration to be tested and in which order. Then for each configuration the model is created, eventually trained, evaluated and its results stored by successively calling the create_model, train_model and evaluate_model methods. If a model contains additional_confs, then a new model will be tested using the previously learned/used code (e.g. a coding scheme and decoders weights are learned GNBP decoder and then the learned code is evaluated under standard BP decoder)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from collections import namedtuple\n",
    "from tools import configurations_product,configurations_list\n",
    "\n",
    "# List of create_model options\n",
    "options = ['n','k','build_dataset','conf','learning_rate','model_index','training_noise_power_db','train_model','G','H','trainable_code','trainable_decoder','additional_confs','name']\n",
    "\n",
    "# Default models configuration (exhaustiv ML, conventional BP and GNBP decoders)\n",
    "ML_eval_conf = configurations_list(options,[[n,k,train_dataset,'ML',None,0,noise_power_training_dbs,False,None,None,False,False,[],\"ML\"]])[0]\n",
    "BP_eval_conf = configurations_list(options,[[n,k,train_dataset,'BP',None,0,noise_power_training_dbs,False,None,None,False,False,[],\"BP\"]])[0]\n",
    "GNBP_eval_conf = configurations_list(options,[[n,k,train_dataset,'GNBP',1e-1,0,noise_power_training_dbs,True,None,None,False,True,[],\"GNBP\"]])[0]\n",
    "\n",
    "# Number of trial for each training\n",
    "n_trials= 2\n",
    "\n",
    "# List of configuration to be tested (following the list of parameters from the create_model method)\n",
    "config_list = [[n,k,train_dataset,'A',1e-1,i,noise_power_training_dbs,True,None,None,True,True,[ML_eval_conf,BP_eval_conf],f'AE_GNBP_{i}'] for i in range(n_trials)]\n",
    "    \n",
    "if use_reference_code:\n",
    "    config_list = config_list + [[n,k,train_dataset,'BP',None,0,noise_power_training_dbs,False,G_sys,H_sys,False,False,[],'BCH_SYS_BP']]                            \\\n",
    "    + [[n,k,train_dataset,'GNBP',1e-1,i,noise_power_training_dbs,True,G_sys,H_sys,False,True,[],f'BCH_SYS_GNBP_{i}'] for i in range(n_trials)]                      \\\n",
    "    + [[n,k,train_dataset,'BP',None,0,noise_power_training_dbs,False,G_sys,H_nsys,False,False,[],'BCH_NSYS_BP']]                                                    \\\n",
    "    + [[n,k,train_dataset,'GNBP',1e-1,i,noise_power_training_dbs,True,G_sys,H_nsys,False,True,[],f'BCH_NSYS_GNBP_{i}'] for i in range(n_trials)]                    \\\n",
    "    + [[n,k,train_dataset,'ML',None,0,noise_power_training_dbs,False,G_sys,H_sys,False,False,[],'BCH_ML']]   \n",
    "\n",
    "# Parse all configuration and test them\n",
    "configurations = configurations_list(options,config_list)\n",
    "for c in configurations:\n",
    "    #print(c)\n",
    "    model = create_model(**c._asdict())\n",
    "    if c.train_model:\n",
    "        train_model(model, models_path, train_dataset, validation_dataset, tensorboard_path,epochs,steps_per_epoch)\n",
    "    (G,H) = model.code_generator(None)\n",
    "    tf.assert_equal(tf.math.floormod(tf.matmul(G,tf.transpose(H)),2), tf.zeros(shape=(k,(n-k))),message=\"Generator and PC matrices are not matched as syndrome matrix (G.H^T) is not equal to 0\")\n",
    "    evaluate_model(summary_ber, summary_bler, summary_bec, summary_blec, summary_bpci_ber, summary_bpci_bler, models_path, model, ebn0_eval_dbs, test_dataset, k, n)\n",
    "    \n",
    "    # If one configuration contains additional configuration load and test them (using the previously used/learned code)\n",
    "    if c.additional_confs != []:\n",
    "        parent_model_name = model.name\n",
    "        for conf in c.additional_confs:           \n",
    "            conf = conf._asdict()\n",
    "            conf['G'] = G\n",
    "            conf['H'] = H\n",
    "            conf['name'] = parent_model_name + \"_\" +conf['name']\n",
    "            Configuration = namedtuple(\"Configuration\", options)\n",
    "            conf = Configuration(**conf)\n",
    "            print(\"Additionnal Conf:\")\n",
    "            model = create_model(**conf._asdict())\n",
    "            if conf.train_model:\n",
    "                train_model(model, models_path, train_dataset, validation_dataset, tensorboard_path, epochs, steps_per_epoch)\n",
    "            evaluate_model(summary_ber, summary_bler, summary_bec, summary_blec, summary_bpci_ber, summary_bpci_bler, models_path, model, ebn0_eval_dbs, test_dataset, k, n)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) 2022 Orange\n",
    "\n",
    "Author: Guillaume Larue <guillaume.larue@orange.com>\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), \n",
    "to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, \n",
    "and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS \n",
    "FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER \n",
    "IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 ('mld')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "a220954edba4173bc5d3bf25ecec93596610de638854d79deb6e8bf2d622dd71"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}