Merge pull request #40 from xsqian/main

add note that depending on the size of the data, this demo can take up to couple of hours to complete.

Author: EdmondIguazio

README.md

@@ -10,11 +10,11 @@ The demo demonstrates two usages of GenAI:
 - Unstructured data generation: Generating audio data with ground truth metadata to evaluate the analysis.
 - Unstructured data analysis: Turning audio calls into text and tabular features.
 
-The demo contains a single [notebook](./call-center-demo.ipynb) that encompasses the entire demo.
+The demo contains two notebooks [notebook 1](./notebook_1_generation.ipynb) and [notebook 2](./notebook_2_analysis.ipynb).
 
 Most of the functions are imported from [MLRun's hub](https://https://www.mlrun.org/hub/), which contains a wide range of functions and modules that can be used for a variety of use cases. See also the [MLRun hub documentation](https://docs.mlrun.org/en/stable/runtimes/load-from-hub.html). All functions used in the demo include links to their source in the hub. All of the python source code is under [/src](./src).
 
-> **⚠️ Important This demo can take an hour to complete when running without GPUs.**
+> **⚠️ Important This demo can take up to couple of hours to complete when running without GPUs.**
 
 ## Prerequisites
 
@@ -26,9 +26,8 @@ This demo uses:
 * [**Vizro**](https://vizro.mckinsey.com/) — To view the call center DB and transcriptions, and to play the generated conversations. 
 * [**MLRun**](https://www.mlrun.org/) — the orchestrator to operationalize the workflow. MLRun 1.9 and higher, Python 3.11, with CPU or GPU.
 * [**SQLAlchemy**](https://www.sqlalchemy.org/) — Manage the MySQL DB of calls, clients and agents. Installed together with MLRun.
-- MySQL database. Installed together with MLRun. (SQLite is not currently supported.)
+- SQLite
 
-<a id="installation"></a>
 ## Installation
 
 This project can run in different development environments:
@@ -54,7 +53,8 @@ Make sure you open the notebooks and select the `mlrun` conda environment
 
 The MLRun service and computation can run locally (minimal setup) or over a remote Kubernetes environment.
 
-If your development environment supports Docker and there are sufficient CPU resources, run:
+
+If your development environment supports Docker and there are sufficient CPU resources (support for Docker setup will be deprecated), run:
 
     make mlrun-docker
 
@@ -72,31 +72,11 @@ in this repo); see [mlrun client setup](https://docs.mlrun.org/en/stable/install
 > Note: You can also use a remote MLRun service (over Kubernetes): instead of starting a local mlrun: 
 edit the [mlrun.env](./mlrun.env) and specify its address and credentials.
 
-### Install SQAlchemy
-
-```
-!pip install SQLAlchemy==2.0.31 pymysql dotenv
-```
-### Setup
-Set the following configuration: choose compute device: CPU or GPU; choose the language of the calls; and whether to skip the calls generation workflow and use pre-generated data. For example:
 
-```
-# True = run with GPU, False = run with CPU
-run_with_gpu = False
-use_sqlite = False
-engine = "remote
-language = "en" # The languages of the calls, es - Spanish, en - English
-skip_calls_generation = False
-```
+#### Setup
 
-#### Setup in Platform McK
-
-Differences between installing on Iguazio cluster and Platform McKinsey:
-- SQLite is supported
 - Set `run_with_gpu = False`, `use_sqlite = True`, `engine = "remote"`.
-- `.env` must include `OPENAI_API_KEY`, `OPENAI_API_BASE`, and `S3_BUCKET_NAME`.
-  * [S3 Bucket]() &mdash; 
-  * `S3_BUCKET_NAME`
+- `.env` must include `OPENAI_API_KEY`, `OPENAI_API_BASE`
 
 ### Configure the tokens and URL
 
@@ -108,55 +88,19 @@ Tokens are required to run the demo end-to-end:
 * [OpenAI ChatGPT](https://chat.openai.com/) &mdash; To generate conversations, two tokens are required:
   * `OPENAI_API_KEY`
   * `OPENAI_API_BASE`
-* [MySQL](https://www.mysql.com/) &mdash; A URL with username and password for collecting the calls into the DB.
-    * `MYSQL_URL`
-    
-> If you want to install mysql using helm chart, use this command:
-> * `helm install -n <"namesapce"> myrelease bitnami/mysql --set auth.rootPassword=sql123 --set auth.database=mlrun_demos --set primary.service.ports.mysql=3111 --set primary.persistence.enabled=false`
-> Example for MYSQL_URL if you use the above command:</br>
-`mysql+pymysql://root:sql123@myrelease-mysql.<"namesapce">.svc.cluster.local:3111/mlrun_demos`
-
-
-### Import
-
-```
-import dotenv
-import os
-import sys
-import mlrun
-dotenv_file = ".env"
-sys.path.insert(0, os.path.abspath("./"))
-
-dotenv.load_dotenv(dotenv_file)
-```
-
-```
-assert not run_with_gpu
-assert os.environ["OPENAI_API_BASE"]
-assert os.environ["OPENAI_API_KEY"]
-```
-
-```
-if not mlrun.mlconf.is_ce_mode():
-    assert os.environ["MYSQL_URL"]
-    use_sqlite = False
-else:
-    use_sqlite = True
-```
 
 ## Demo flow
 
 1. Create the project
-- **Notebook**: [call-center-demo.ipynb](call-center-demo.ipynb)
+- **Notebook**: [notebook_1_generation.ipynb](notebook_1_generation.ipynb)
 - **Description**: 
 - **Key steps**: Create the MLRun project. 
 - **Key files**:
-  - [project.yaml](./project.yaml)
   - [project_setup.py](./project_setup.py)
 
 2. Generate the call data
 
-- **Notebook**: [call-center-demo.ipynb](call-center-demo.ipynb)
+- **Notebook**: [notebook_1_generation.ipynb](notebook_1_generation.ipynb)
 - **Description**: Generate the call data. (You can choose to skip this step ans use call data that is already generated and available in the demo.)
 - **Key steps**: To generate data, run: Agents & clients data generator, Insert agents & clients data to DB, Get agents & clients from DB, Conversation generation, Text to Audio, and Batch Creation. and Batch creation. Then run the workflow.
 
@@ -170,7 +114,7 @@ else:
 
 3. Calls analysis
 
-- **Notebook**: [call-center-demo.ipynb](call-center-demo.ipynb)
+- **Notebook**: [notebook_2_analysis.ipynb](notebook_2_analysis.ipynb)
 - **Description**: Insert the call data to the DB, use diarization to analyze when each person is speaking, transcribe and translate the calls into text and save them as text files, recognice and remove any PII
 , anaylze text (call center conversation) with an LLM, postprocess the LLM's answers before updating them into the DB. Then run the all analysis workflow.
 - **Key steps**: Insert the calls data to the DB, perform speech diarization, transcribe, recognize PII, analysis. Then run the workflow.
@@ -187,5 +131,5 @@ else:
 
 4. View the data
 
-- **Notebook**: [call-center-demo.ipynb](call-center-demo.ipynb)
+- **Notebook**: [notebook_2_analysis.ipynb](notebook_2_analysis.ipynb)
 - **Description**: View the data and features, as they are collected, in the MLRun UI. Deploy [Vizro](https://vizro.mckinsey.com/) to visualize the data in the DB.

call-center-demo.ipynb

@@ -22,10 +22,15 @@
     "## Table of contents:\n",
     "\n",
     "1. [Create the project](#create_the_project)\n",
-    "2. [Generate the call data](#generate_the_call_data)\n",
-    "3. [Calls analysis](#calls_analysis)\n",
-    "4. [View the data](#view_the_data)\n",
-    "5. [Future work](#future_work)"
+    "2. [Generate the call data](#generate_the_call_data)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "481122ad-a858-4edf-8c63-9fbf3c8df518",
+   "metadata": {},
+   "source": [
+    "> **⚠️ Important Depending on the size of the data, this demo can take up to couple of hours to complete.**"
    ]
   },
   {
@@ -81,7 +86,7 @@
    "id": "c1a3e034-151f-46d0-b6f7-38876c113a8b",
    "metadata": {},
    "source": [
-    "#### Setup in Iguazio Cluster"
+    "#### Setup"
    ]
   },
   {
@@ -91,26 +96,6 @@
    "source": [
     "- This demo is limited to run Python 3.11, with CPU, and run the pipeline with `engine = \"remote\"`.\n",
     "- GPU is not supported at the moment.\n",
-    "- Need to setup a mysql database for the demo. sqlite is not supported now.\n",
-    "- Please set `run_with_gpu = False`, `engine = \"remote\"`\n",
-    "- .env include OPENAI_API_KEY, OPENAI_API_BASE"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6c44bb2b-5ded-49d6-ae4d-7fa9ca67d1e4",
-   "metadata": {},
-   "source": [
-    "#### Setup in Platform McK"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "05e3b4b8-d370-4fcf-944b-e8f3b478fa24",
-   "metadata": {},
-   "source": [
-    "- GPU is not supported at the moment.\n",
-    "- sqlite is supported.\n",
     "- Please set `run_with_gpu = False`, `engine = \"remote\"`\n",
     "- .env include OPENAI_API_KEY, OPENAI_API_BASE"
    ]

notebook_1_generation.ipynb

@@ -29,40 +29,29 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0537e242-f14b-4d68-a8ca-b45357849f4c",
+   "id": "c01ac25f-2ba8-4d3a-a22d-975178ca9e53",
    "metadata": {},
    "source": [
-    "___\n",
-    "<a id=\"get_the_project\"></a>\n",
-    "## Get the project "
+    "> **⚠️ Important Depending on the size of the data, this demo can take up to couple of hours to complete.**"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "330ea34f-2d34-472c-995f-9f171afb03cf",
-   "metadata": {},
-   "source": [
-    "- This demo is limited to Python 3.11, with CPU, and run the pipeline with `engine = \"remote\"`.\n",
-    "- GPU is not supported at the moment.\n",
-    "- Please set `run_with_gpu = False`, `engine = \"remote\"`\n",
-    "- .env include OPENAI_API_KEY, OPENAI_API_BASE"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6c44bb2b-5ded-49d6-ae4d-7fa9ca67d1e4",
+   "id": "0537e242-f14b-4d68-a8ca-b45357849f4c",
    "metadata": {},
    "source": [
-    "#### Setup in Platform McK"
+    "___\n",
+    "<a id=\"get_the_project\"></a>\n",
+    "## 1. Get the project "
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "05e3b4b8-d370-4fcf-944b-e8f3b478fa24",
+   "id": "330ea34f-2d34-472c-995f-9f171afb03cf",
    "metadata": {},
    "source": [
+    "- This demo is limited to Python 3.11, with CPU, and run the pipeline with `engine = \"remote\"`.\n",
     "- GPU is not supported at the moment.\n",
-    "- sqlite is supported.\n",
     "- Please set `run_with_gpu = False`, `engine = \"remote\"`\n",
     "- .env include OPENAI_API_KEY, OPENAI_API_BASE"
    ]
@@ -72,7 +61,7 @@
    "id": "b5eb3156-4dba-4ef7-a406-13e89772e700",
    "metadata": {},
    "source": [
-    "### Fill the tokens and URL\n",
+    "### 1.1 Fill the tokens and URL\n",
     "\n",
     "> **⚠️ Important** Please fill the following variables in your `.env` file.\n",
     "\n",
@@ -130,7 +119,7 @@
    "id": "1ea33fee-ec95-48e3-aae8-9247ae182481",
    "metadata": {},
    "source": [
-    "###  Get the current project\n",
+    "### 1.2 Get the current project\n",
     "\n",
     "The MLRun project is created by running the function [`mlrun.get_or_create_project`](https://docs.mlrun.org/en/latest/api/mlrun.projects.html#mlrun.projects.get_or_create_project). This creates the project (or loads it if previously created) and sets it up automatically according to the [project_setup.py](./project_setup.py) file located in this repo. \n",
     "\n",
@@ -181,7 +170,7 @@
    "source": [
     "___\n",
     "<a id=\"calls_analysis\"></a>\n",
-    "## 3. Calls analysis\n",
+    "## 2. Calls analysis\n",
     "\n",
     "The workflow includes multiple steps for which all of the main functions are imported from the **[MLRun Function Hub](https://www.mlrun.org/hub/)**. You can see each hub function's docstring, code, and example, by clicking the function name in the following list:\n",
     "\n",
@@ -227,7 +216,7 @@
    "id": "c68f2edd-0c0b-403a-8fc6-156b299dc8f1",
    "metadata": {},
    "source": [
-    "### 3.1. Run the workflow\n",
+    "### 2.1. Run the workflow\n",
     "\n",
     "Now, run the workflow using the following parameters:\n",
     "* `batch: str` &mdash; Path to the dataframe artifact that represents the batch to analyze. \n",
@@ -303,7 +292,7 @@
    "source": [
     "___\n",
     "<a id=\"view_the_data\"></a>\n",
-    "## 4. View the data\n",
+    "## 3. View the data\n",
     "\n",
     "While the workflow is running, you can view the data and features as they are collected.\n",
     "\n",
@@ -375,7 +364,7 @@
    "source": [
     "___\n",
     "<a id=\"future_work\"></a>\n",
-    "## 5. Future work\n",
+    "## 4. Future work\n",
     "\n",
     "This demo is a proof of concept for LLM's feature-extraction capabilities, while using MLRun for the orchestration from development to production. The demo continues to be developed. You are welcome to track and develop it with us:\n",
     "\n",