notebooks/242-freevc-voice-conversion/242-freevc-voice-conversion.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High-Quality Text-Free One-Shot Voice Conversion with FreeVC and OpenVINO™\n",
    "[FreeVC](https://github.com/OlaWod/FreeVC) allows alter the voice of a source speaker to a target style, while keeping the linguistic content unchanged, without text annotation.\n",
    "\n",
    "Figure bellow illustrates model architecture of FreeVC for inference. In this notebook we concentrate only on inference part. There are three main parts: Prior Encoder, Speaker Encoder and Decoder. The prior encoder contains a WavLM model, a bottleneck extractor and a normalizing flow. Detailed information is available in this [paper](https://arxiv.org/abs/2210.15418).\n",
    "\n",
    "![Inference](https://github.com/OlaWod/FreeVC/blob/main/resources/infer.png?raw=true)\n",
    "\n",
    "[**image_source*](https://github.com/OlaWod/FreeVC)\n",
    "\n",
    "FreeVC suggests only command line interface to use and only with CUDA. In this notebook it shows how to use FreeVC in Python and without CUDA devices. It consists of the following steps:\n",
    "\n",
    "- Download and prepare models.\n",
    "- Inference.\n",
    "- Convert models to OpenVINO Intermediate Representation.\n",
    "- Inference using only OpenVINO's IR models.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of content:\n",
    "- [Pre-requisites](#Pre-requisites-Uparrow)\n",
    "- [Imports and settings](#Imports-and-settings-Uparrow)\n",
    "- [Convert Modes to OpenVINO Intermediate Representation](#Convert-Modes-to-OpenVINO-Intermediate-Representation-Uparrow)\n",
    "    - [Convert Prior Encoder.](#Convert-Prior-Encoder.-Uparrow)\n",
    "    - [Convert `SpeakerEncoder`](#Convert-`SpeakerEncoder`-Uparrow)\n",
    "    - [Convert Decoder](#Convert-Decoder-Uparrow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-requisites [$\\Uparrow$](#Table-of-content:)\n",
    "This steps can be done manually or will be performed automatically during the execution of the notebook, but in minimum necessary scope.\n",
    "1. Clone this repo: git clone https://github.com/OlaWod/FreeVC.git.\n",
    "2. Download [WavLM-Large](https://github.com/microsoft/unilm/tree/master/wavlm) and put it under directory `FreeVC/wavlm/`.\n",
    "3. You can download the [VCTK](https://datashare.ed.ac.uk/handle/10283/3443) dataset. For this example we download only two of them from [Hugging Face FreeVC example](https://huggingface.co/spaces/OlaWod/FreeVC/tree/main).\n",
    "4. Download [pretrained models](https://1drv.ms/u/s!AnvukVnlQ3ZTx1rjrOZ2abCwuBAh?e=UlhRR5) and put it under directory 'checkpoints' (for current example only `freevc.pth` are required)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install extra requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q \"librosa>=0.8.1\"\n",
    "%pip install -q \"webrtcvad==2.0.10\"\n",
    "%pip install -q \"openvino>=2023.1.0\"\n",
    "%pip install -q gradio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if FreeVC is installed and append its path to `sys.path`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import sys\n",
    "\n",
    "\n",
    "free_vc_repo = 'FreeVC'\n",
    "if not Path(free_vc_repo).exists():\n",
    "    !git clone https://github.com/OlaWod/FreeVC.git\n",
    "\n",
    "sys.path.append(free_vc_repo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file\n",
    "\n",
    "wavlm_large_dir_path = Path('FreeVC/wavlm')\n",
    "wavlm_large_path = wavlm_large_dir_path / 'WavLM-Large.pt'\n",
    "\n",
    "if not wavlm_large_path.exists():\n",
    "    download_file(\n",
    "        'https://valle.blob.core.windows.net/share/wavlm/WavLM-Large.pt?sv=2020-08-04&st=2023-03-01T07%3A51%3A05Z&se=2033-03-02T07%3A51%3A00Z&sr=c&sp=rl&sig=QJXmSJG9DbMKf48UDIU1MfzIro8HQOf3sqlNXiflY1I%3D',\n",
    "        directory=wavlm_large_dir_path\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freevc_chpt_dir = Path('checkpoints')\n",
    "freevc_chpt_name = 'freevc.pth'\n",
    "freevc_chpt_path = freevc_chpt_dir / freevc_chpt_name\n",
    "\n",
    "if not freevc_chpt_path.exists():\n",
    "    download_file(\n",
    "        f'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/freevc/{freevc_chpt_name}',\n",
    "        directory=freevc_chpt_dir\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "audio1_name = 'p225_001.wav'\n",
    "audio1_url = f'https://huggingface.co/spaces/OlaWod/FreeVC/resolve/main/{audio1_name}'\n",
    "audio2_name = 'p226_002.wav'\n",
    "audio2_url = f'https://huggingface.co/spaces/OlaWod/FreeVC/resolve/main/{audio2_name}'\n",
    "\n",
    "if not Path(audio1_name).exists():\n",
    "    download_file(audio1_url)\n",
    "\n",
    "if not Path(audio2_name).exists():\n",
    "    download_file(audio2_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and settings [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import time\n",
    "\n",
    "import librosa\n",
    "import numpy as np\n",
    "import torch\n",
    "from scipy.io.wavfile import write\n",
    "from tqdm import tqdm\n",
    "\n",
    "import openvino as ov\n",
    "\n",
    "import utils\n",
    "from models import SynthesizerTrn\n",
    "from speaker_encoder.voice_encoder import SpeakerEncoder\n",
    "from wavlm import WavLM, WavLMConfig\n",
    "\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.CRITICAL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Redefine function `get_model` from `utils` to exclude CUDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cmodel():\n",
    "    checkpoint = torch.load(wavlm_large_path)\n",
    "    cfg = WavLMConfig(checkpoint['cfg'])\n",
    "    cmodel = WavLM(cfg)\n",
    "    cmodel.load_state_dict(checkpoint['model'])\n",
    "    cmodel.eval()\n",
    "\n",
    "    return cmodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hps = utils.get_hparams_from_file('FreeVC/configs/freevc.json')\n",
    "os.makedirs('outputs/freevc', exist_ok=True)\n",
    "\n",
    "net_g = SynthesizerTrn(\n",
    "    hps.data.filter_length // 2 + 1,\n",
    "    hps.train.segment_size // hps.data.hop_length,\n",
    "    **hps.model\n",
    ")\n",
    "\n",
    "utils.load_checkpoint(freevc_chpt_path, net_g, optimizer=None, strict=True)\n",
    "cmodel = get_cmodel()\n",
    "smodel = SpeakerEncoder('FreeVC/speaker_encoder/ckpt/pretrained_bak_5805000.pt', device='cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading dataset settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "srcs = [audio1_name, audio2_name]\n",
    "tgts = [audio2_name, audio1_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    for line in tqdm(zip(srcs, tgts)):\n",
    "        src, tgt = line\n",
    "        # tgt\n",
    "        wav_tgt, _ = librosa.load(tgt, sr=hps.data.sampling_rate)\n",
    "        wav_tgt, _ = librosa.effects.trim(wav_tgt, top_db=20)\n",
    "\n",
    "        g_tgt = smodel.embed_utterance(wav_tgt)\n",
    "        g_tgt = torch.from_numpy(g_tgt).unsqueeze(0)\n",
    "\n",
    "        # src\n",
    "        wav_src, _ = librosa.load(src, sr=hps.data.sampling_rate)\n",
    "        wav_src = torch.from_numpy(wav_src).unsqueeze(0)\n",
    "\n",
    "        c = utils.get_content(cmodel, wav_src)\n",
    "\n",
    "        tgt_audio = net_g.infer(c, g=g_tgt)\n",
    "        tgt_audio = tgt_audio[0][0].data.cpu().float().numpy()\n",
    "\n",
    "        timestamp = time.strftime(\"%m-%d_%H-%M\", time.localtime())\n",
    "        write(os.path.join('outputs/freevc', \"{}.wav\".format(timestamp)), hps.data.sampling_rate,\n",
    "              tgt_audio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Result audio files should be available in 'outputs/freevc'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert Modes to OpenVINO Intermediate Representation [$\\Uparrow$](#Table-of-content:)\n",
    "Convert each model to ONNX format and then use the model conversion Python API to convert the ONNX model to OpenVINO IR, with FP16 precision. The `ov.convert_model` function accepts the path to a model and returns the OpenVINO Model class instance which represents this model. The obtained model is ready to use and to be loaded on a device using `compile_model` or can be saved on a disk using the `ov.save_model` function. The `read_model` method loads a saved model from a disk.\n",
    "For more information about model conversion, see this [page](https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html).\n",
    "### Convert Prior Encoder. [$\\Uparrow$](#Table-of-content:)\n",
    "First we convert WavLM model, as a part of Convert Prior Encoder, to the ONNX format, then to OpenVINO's IR format. We keep the original name of the model in code: `cmodel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define forward as extract_features for compatibility\n",
    "cmodel.forward = cmodel.extract_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert `cmodel` to ONNX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTPUT_DIR = Path(\"output\")\n",
    "BASE_MODEL_NAME = \"cmodel\"\n",
    "\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "onnx_cmodel_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"_fp32\")).with_suffix(\".onnx\")\n",
    "ir_cmodel_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"_ir\")).with_suffix(\".xml\")\n",
    "\n",
    "length = 32000\n",
    "input_shape = (1, length)\n",
    "\n",
    "input_names = ['input']\n",
    "output_names = ['output']\n",
    "dummy_input = torch.randn(1, length)\n",
    "dynamic_axes = {\n",
    "    'input': {1: 'length'},\n",
    "    'output': {1: 'out_length'}\n",
    "}\n",
    "\n",
    "torch.onnx.export(\n",
    "    cmodel,\n",
    "    dummy_input,\n",
    "    onnx_cmodel_path,\n",
    "    input_names=input_names,\n",
    "    output_names=output_names,\n",
    "    dynamic_axes=dynamic_axes\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting to OpenVINO's IR format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "core = ov.Core()\n",
    "\n",
    "if not ir_cmodel_path.exists():\n",
    "    ir_cmodel = ov.convert_model(onnx_cmodel_path)\n",
    "    ov.save_model(ir_cmodel, ir_cmodel_path)\n",
    "else:\n",
    "    ir_cmodel = core.read_model(ir_cmodel_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select device from dropdown list for running inference using OpenVINO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices + [\"AUTO\"],\n",
    "    value='AUTO',\n",
    "    description='Device:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "compiled_cmodel = core.compile_model(ir_cmodel, device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert `SpeakerEncoder` [$\\Uparrow$](#Table-of-content:)\n",
    "Converting to ONNX format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTPUT_DIR = Path(\"output\")\n",
    "BASE_MODEL_NAME = \"smodel\"\n",
    "\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "onnx_smodel_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"_fp32\")).with_suffix(\".onnx\")\n",
    "ir_smodel_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"ir\")).with_suffix(\".xml\")\n",
    "\n",
    "\n",
    "length = 32000\n",
    "\n",
    "input_names = ['input']\n",
    "output_names = ['output']\n",
    "dummy_input = torch.randn(1, length, 40)\n",
    "dynamic_axes = {\n",
    "    'input': {\n",
    "        0: 'branch_size',\n",
    "        1: 'length'\n",
    "    },\n",
    "    'output': {1: 'out_length'}\n",
    "}\n",
    "\n",
    "torch.onnx.export(smodel, dummy_input, onnx_smodel_path, input_names=input_names, output_names=output_names, dynamic_axes=dynamic_axes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting to OpenVINO's IR format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not ir_smodel_path.exists():\n",
    "    ir_smodel = ov.convert_model(onnx_smodel_path)\n",
    "    ov.save_model(ir_smodel, ir_smodel_path)\n",
    "else:\n",
    "    ir_smodel = core.read_model(ir_smodel_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For preparing input for inference, we should define helper functions based on `speaker_encoder.voice_encoder.SpeakerEncoder` class methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from speaker_encoder.hparams import sampling_rate, mel_window_step, partials_n_frames\n",
    "from speaker_encoder import audio\n",
    "\n",
    "\n",
    "def compute_partial_slices(n_samples: int, rate, min_coverage):\n",
    "    \"\"\"\n",
    "    Computes where to split an utterance waveform and its corresponding mel spectrogram to\n",
    "    obtain partial utterances of <partials_n_frames> each. Both the waveform and the\n",
    "    mel spectrogram slices are returned, so as to make each partial utterance waveform\n",
    "    correspond to its spectrogram.\n",
    "\n",
    "    The returned ranges may be indexing further than the length of the waveform. It is\n",
    "    recommended that you pad the waveform with zeros up to wav_slices[-1].stop.\n",
    "\n",
    "    :param n_samples: the number of samples in the waveform\n",
    "    :param rate: how many partial utterances should occur per second. Partial utterances must\n",
    "    cover the span of the entire utterance, thus the rate should not be lower than the inverse\n",
    "    of the duration of a partial utterance. By default, partial utterances are 1.6s long and\n",
    "    the minimum rate is thus 0.625.\n",
    "    :param min_coverage: when reaching the last partial utterance, it may or may not have\n",
    "    enough frames. If at least <min_pad_coverage> of <partials_n_frames> are present,\n",
    "    then the last partial utterance will be considered by zero-padding the audio. Otherwise,\n",
    "    it will be discarded. If there aren't enough frames for one partial utterance,\n",
    "    this parameter is ignored so that the function always returns at least one slice.\n",
    "    :return: the waveform slices and mel spectrogram slices as lists of array slices. Index\n",
    "    respectively the waveform and the mel spectrogram with these slices to obtain the partial\n",
    "    utterances.\n",
    "    \"\"\"\n",
    "    assert 0 < min_coverage <= 1\n",
    "\n",
    "    # Compute how many frames separate two partial utterances\n",
    "    samples_per_frame = int((sampling_rate * mel_window_step / 1000))\n",
    "    n_frames = int(np.ceil((n_samples + 1) / samples_per_frame))\n",
    "    frame_step = int(np.round((sampling_rate / rate) / samples_per_frame))\n",
    "    assert 0 < frame_step, \"The rate is too high\"\n",
    "    assert frame_step <= partials_n_frames, \"The rate is too low, it should be %f at least\" % \\\n",
    "        (sampling_rate / (samples_per_frame * partials_n_frames))\n",
    "\n",
    "    # Compute the slices\n",
    "    wav_slices, mel_slices = [], []\n",
    "    steps = max(1, n_frames - partials_n_frames + frame_step + 1)\n",
    "    for i in range(0, steps, frame_step):\n",
    "        mel_range = np.array([i, i + partials_n_frames])\n",
    "        wav_range = mel_range * samples_per_frame\n",
    "        mel_slices.append(slice(*mel_range))\n",
    "        wav_slices.append(slice(*wav_range))\n",
    "\n",
    "    # Evaluate whether extra padding is warranted or not\n",
    "    last_wav_range = wav_slices[-1]\n",
    "    coverage = (n_samples - last_wav_range.start) / (last_wav_range.stop - last_wav_range.start)\n",
    "    if coverage < min_coverage and len(mel_slices) > 1:\n",
    "        mel_slices = mel_slices[:-1]\n",
    "        wav_slices = wav_slices[:-1]\n",
    "\n",
    "    return wav_slices, mel_slices\n",
    "\n",
    "\n",
    "def embed_utterance(wav: np.ndarray, smodel: ov.CompiledModel, return_partials=False, rate=1.3, min_coverage=0.75):\n",
    "    \"\"\"\n",
    "    Computes an embedding for a single utterance. The utterance is divided in partial\n",
    "    utterances and an embedding is computed for each. The complete utterance embedding is the\n",
    "    L2-normed average embedding of the partial utterances.\n",
    "\n",
    "    :param wav: a preprocessed utterance waveform as a numpy array of float32\n",
    "    :param smodel: compiled speaker encoder model.\n",
    "    :param return_partials: if True, the partial embeddings will also be returned along with\n",
    "    the wav slices corresponding to each partial utterance.\n",
    "    :param rate: how many partial utterances should occur per second. Partial utterances must\n",
    "    cover the span of the entire utterance, thus the rate should not be lower than the inverse\n",
    "    of the duration of a partial utterance. By default, partial utterances are 1.6s long and\n",
    "    the minimum rate is thus 0.625.\n",
    "    :param min_coverage: when reaching the last partial utterance, it may or may not have\n",
    "    enough frames. If at least <min_pad_coverage> of <partials_n_frames> are present,\n",
    "    then the last partial utterance will be considered by zero-padding the audio. Otherwise,\n",
    "    it will be discarded. If there aren't enough frames for one partial utterance,\n",
    "    this parameter is ignored so that the function always returns at least one slice.\n",
    "    :return: the embedding as a numpy array of float32 of shape (model_embedding_size,). If\n",
    "    <return_partials> is True, the partial utterances as a numpy array of float32 of shape\n",
    "    (n_partials, model_embedding_size) and the wav partials as a list of slices will also be\n",
    "    returned.\n",
    "    \"\"\"\n",
    "    # Compute where to split the utterance into partials and pad the waveform with zeros if\n",
    "    # the partial utterances cover a larger range.\n",
    "    wav_slices, mel_slices = compute_partial_slices(len(wav), rate, min_coverage)\n",
    "    max_wave_length = wav_slices[-1].stop\n",
    "    if max_wave_length >= len(wav):\n",
    "        wav = np.pad(wav, (0, max_wave_length - len(wav)), \"constant\")\n",
    "\n",
    "    # Split the utterance into partials and forward them through the model\n",
    "    mel = audio.wav_to_mel_spectrogram(wav)\n",
    "    mels = np.array([mel[s] for s in mel_slices])\n",
    "    with torch.no_grad():\n",
    "        mels = torch.from_numpy(mels).to(torch.device('cpu'))\n",
    "        output_layer = smodel.output(0)\n",
    "        partial_embeds = smodel(mels)[output_layer]\n",
    "\n",
    "    # Compute the utterance embedding from the partial embeddings\n",
    "    raw_embed = np.mean(partial_embeds, axis=0)\n",
    "    embed = raw_embed / np.linalg.norm(raw_embed, 2)\n",
    "\n",
    "    if return_partials:\n",
    "        return embed, partial_embeds, wav_slices\n",
    "    return embed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select device from dropdown list for running inference using OpenVINO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then compile model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compiled_smodel = core.compile_model(ir_smodel, device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert Decoder [$\\Uparrow$](#Table-of-content:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same way export `SynthesizerTrn` model, that implements decoder function, to ONNX format and convert it to OpenVINO IR format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTPUT_DIR = Path(\"output\")\n",
    "BASE_MODEL_NAME = \"net_g\"\n",
    "onnx_net_g_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"_fp32\")).with_suffix(\".onnx\")\n",
    "ir_net_g_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"ir\")).with_suffix(\".xml\")\n",
    "\n",
    "dummy_input_1 = torch.randn(1, 1024, 81)\n",
    "dummy_input_2 = torch.randn(1, 256)\n",
    "\n",
    "input_names = ['input1', 'input2']\n",
    "output_names = ['output']\n",
    "dynamic_axes = {\n",
    "    'input1': {\n",
    "        0: 'branch_size',\n",
    "        2: 'length'\n",
    "    },\n",
    "    'input2': {\n",
    "        0: 'branch_size',\n",
    "    },\n",
    "    'output': {1: 'out_length'}\n",
    "}\n",
    "\n",
    "# define forward as infer\n",
    "net_g.forward = net_g.infer\n",
    "\n",
    "torch.onnx.export(net_g, (dummy_input_1, dummy_input_2), onnx_net_g_path, input_names=input_names, output_names=output_names, dynamic_axes=dynamic_axes)\n",
    "\n",
    "if not ir_net_g_path.exists():\n",
    "    ir_net_g_model = ov.convert_model(onnx_net_g_path)\n",
    "    ov.save_model(ir_net_g_model, ir_net_g_path)\n",
    "else:\n",
    "    ir_net_g_model = core.read_model(ir_net_g_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select device from dropdown list for running inference using OpenVINO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compiled_ir_net_g_model = core.compile_model(ir_net_g_model, device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define function for synthesizing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def synthesize_audio(src, tgt):\n",
    "    wav_tgt, _ = librosa.load(tgt, sr=hps.data.sampling_rate)\n",
    "    wav_tgt, _ = librosa.effects.trim(wav_tgt, top_db=20)\n",
    "\n",
    "    g_tgt = embed_utterance(wav_tgt, compiled_smodel)\n",
    "    g_tgt = torch.from_numpy(g_tgt).unsqueeze(0)\n",
    "\n",
    "    # src\n",
    "    wav_src, _ = librosa.load(src, sr=hps.data.sampling_rate)\n",
    "    wav_src = np.expand_dims(wav_src, axis=0)\n",
    "\n",
    "    output_layer = compiled_cmodel.output(0)\n",
    "    c = compiled_cmodel(wav_src)[output_layer]\n",
    "    c = c.transpose((0, 2, 1))\n",
    "\n",
    "    output_layer = compiled_ir_net_g_model.output(0)\n",
    "    tgt_audio = compiled_ir_net_g_model((c, g_tgt))[output_layer]\n",
    "    tgt_audio = tgt_audio[0][0]\n",
    "\n",
    "    return tgt_audio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can check inference using only IR models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_wav_names = []\n",
    "\n",
    "with torch.no_grad():\n",
    "    for line in tqdm(zip(srcs, tgts)):\n",
    "        src, tgt = line\n",
    "\n",
    "        output_audio = synthesize_audio(src, tgt)\n",
    "\n",
    "        timestamp = time.strftime(\"%m-%d_%H-%M\", time.localtime())\n",
    "        result_name = f'{timestamp}.wav'\n",
    "        result_wav_names.append(result_name)\n",
    "        write(\n",
    "            os.path.join('outputs/freevc', result_name),\n",
    "            hps.data.sampling_rate,\n",
    "            output_audio\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Result audio files should be available in 'outputs/freevc' and you can check them and compare with generated earlier.\n",
    "Below one of the results presents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Source audio (source of text):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython.display as ipd\n",
    "ipd.Audio(srcs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Target audio (source of voice):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ipd.Audio(tgts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Result audio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ipd.Audio(f'outputs/freevc/{result_wav_names[0]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, you can use your own audio file. Just upload them and use for inference. Use rate corresponding to the value of `hps.data.sampling_rate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gradio as gr\n",
    "\n",
    "\n",
    "audio1 = gr.inputs.Audio(label=\"Source Audio\", type='filepath')\n",
    "audio2 = gr.inputs.Audio(label=\"Reference Audio\", type='filepath')\n",
    "outputs = gr.outputs.Audio(label=\"Output Audio\", type='filepath')\n",
    "examples = [[audio1_name, audio2_name]]\n",
    "\n",
    "title = 'FreeVC with Gradio'\n",
    "description = 'Gradio Demo for FreeVC and OpenVINO™. Upload a source audio and a reference audio, then click the \"Submit\" button to inference.'\n",
    "\n",
    "\n",
    "def infer(src, tgt):\n",
    "    output_audio = synthesize_audio(src, tgt)\n",
    "\n",
    "    timestamp = time.strftime(\"%m-%d_%H-%M\", time.localtime())\n",
    "    result_name = f'{timestamp}.wav'\n",
    "    write(result_name, hps.data.sampling_rate, output_audio)\n",
    "\n",
    "    return result_name\n",
    "\n",
    "\n",
    "iface = gr.Interface(infer, [audio1, audio2], outputs, title=title, description=description, examples=examples)\n",
    "iface.launch()\n",
    "# if you are launching remotely, specify server_name and server_port\n",
    "# iface.launch(server_name='your server name', server_port='server port in int')\n",
    "# if you have any issue to launch on your platform, you can pass share=True to launch method:\n",
    "# iface.launch(share=True)\n",
    "# it creates a publicly shareable link for the interface. Read more in the docs: https://gradio.app/docs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iface.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}