Merge branch 'main' into dependabot/pip/app/backend/langchain-experimental-0.0.52

Author: dayland

Makefile

@@ -24,7 +24,7 @@ build-containers: extract-env
 infrastructure: check-subscription ## Deploy infrastructure
 	@./scripts/inf-create.sh
 
-extract-env: extract-env-debug-webapp extract-env-debug-functions ## Extract infrastructure.env file from BICEP output
+extract-env: extract-env-debug-webapp extract-env-debug-functions ## Extract infrastructure.env file from Terraform output
 	 @./scripts/json-to-env.sh < inf_output.json > ./scripts/environments/infrastructure.env
 
 deploy-webapp: extract-env ## Deploys the web app code to Azure App Service
@@ -39,10 +39,10 @@ deploy-enrichments: extract-env ## Deploys the web app code to Azure App Service
 deploy-search-indexes: extract-env ## Deploy search indexes
 	@./scripts/deploy-search-indexes.sh
 
-extract-env-debug-webapp: ## Extract infrastructure.debug.env file from BICEP output
+extract-env-debug-webapp: ## Extract infrastructure.debug.env file from Terraform output
 	@./scripts/json-to-env.webapp.debug.sh < inf_output.json > ./scripts/environments/infrastructure.debug.env
 
-extract-env-debug-functions: ## Extract local.settings.json to debug functions from BICEP output
+extract-env-debug-functions: ## Extract local.settings.json to debug functions from Terraform output
 	@./scripts/json-to-env.function.debug.sh < inf_output.json > ./functions/local.settings.json
 
 # Utils (used by other Makefile rules)
@@ -64,5 +64,5 @@ destroy-inf: check-subscription
 functional-tests: extract-env ## Run functional tests to check the processing pipeline is working
 	@./scripts/functional-tests.sh	
 
-run-migration: ## Migrate from bicep to terraform
+run-migration: ## Migrate from BICEP to Terraform
 	python ./scripts/merge-databases.py

README.md

@@ -168,7 +168,7 @@ docs/deployment/ | Detailed documentation on how to deploy and start using Infor
 docs/features/ | Detailed documentation of specific features and development level configuration for Information Assistant.
 docs/ | Other supporting documentation that is primarily linked to from the other markdown files.
 functions/ | The pipeline of Azure Functions that handle the document extraction and chunking as well as the custom CosmosDB logging.
-infra/ | The BICEP scripts that deploy the entire IA Accelerator. The overall accelerator is orchestrated via the `main.bicep` file but most of the resource deployments are modularized under the **core** folder.
+infra/ | The Terraform scripts that deploy the entire IA Accelerator. The overall accelerator is orchestrated via the `main.tf` file but most of the resource deployments are modularized under the **core** folder.
 pipelines/ | Azure DevOps pipelines that can be used to enable CI/CD deployments of the accelerator.
 scripts/environments/ | Deployment configuration files. This is where all external configuration values will be set.
 scripts/ | Supporting scripts that perform the various deployment tasks such as infrastructure deployment, Azure WebApp and Function deployments, building of the webapp and functions source code, etc. These scripts align to the available commands in the `Makefile`.

docs/deployment/autoscale_sku.md

@@ -7,11 +7,9 @@ You may find better settings to fit your needs. This document explains how this
 
 ### Overview
 
-The Azure Functions Service Plan Autoscale settings are defined in the file located at `/infra/core/host/funcserviceplan.bicep`. These settings enable automatic scaling of the Azure Functions Service Plan based on CPU usage metrics.
+The Azure Functions Service Plan Autoscale settings are defined in the file located at `/infra/core/host/functions/functions.tf`. These settings enable automatic scaling of the Azure Functions Service Plan based on CPU usage metrics.
 
-
-
- **File Location:** `/infra/core/host/funcserviceplan.bicep`
+ **File Location:** `/infra/core/host/functions/functions.tf`
 
 #### Scaling Rules
 
@@ -29,16 +27,15 @@ The Azure Functions Service Plan Autoscale settings are defined in the file loca
    - **Time Window:** `5 minutes`
    - **Scaling Action:** Decrease capacity by `2` with a cooldown of `2 minutes`.
 
-
 ## App Service Plan Autoscale for Enrichment App
 
 ### Overview
 
-The App Service Plan Autoscale settings for the enrichment app are defined in the file located at `/infra/core/host.enrichmentappserviceplan.bicep`. These settings enable automatic scaling of the App Service Plan based on CPU usage metrics.
+The App Service Plan Autoscale settings for the enrichment app are defined in the file located at `/infra/core/host/enrichmentapp/enrichmentapp.tf`. These settings enable automatic scaling of the App Service Plan based on CPU usage metrics.
 
 ### Key Parameters
 
-**File Location:** `/infra/core/host.enrichmentappserviceplan.bicep`
+**File Location:** `/infra/core/host/enrichmentapp/enrichmentapp.tf`
 
 #### Scaling Rules
 
@@ -58,86 +55,82 @@ The App Service Plan Autoscale settings for the enrichment app are defined in th
 
 ### Customization
 
-To customize the App Service Plan Autoscale settings, modify the parameters mentioned above in the specified Bicep file. And Run the `make infrastructure` command.
-
-
+To customize the App Service Plan Autoscale settings, modify the parameters mentioned above in the specified terraform files. And Run the `make infrastructure` command.
 
 # SKU Settings Documentation
 
 ### Overview
 
-The SKU settings for all Service Plans are defined in the file located at `/infra/main.bicep`.  The SKU (Stock Keeping Unit) represents the pricing tier or plan for your App Service. It defines the performance, features, and capacity of the App Service. 
+The SKU settings for all Service Plans are defined in the file located at `/infra/variables.tf`.  The SKU (Stock Keeping Unit) represents the pricing tier or plan for your App Service. It defines the performance, features, and capacity of the App Service. 
 More information can be found [here.](https://azure.microsoft.com/en-us/pricing/details/app-service/windows/#purchase-options)
 
 ## Web App Service Plan SKU
 
+**File Location:** `/infra/variables.tf`
 
-**File Location:** `/infra/main.bicep`
-
-#### SKU Settings
-
-- **Name:** `S1`
-- **Capacity:** `3`
+### SKU Settings
 
+- **appServiceSkuSize** `S1`
+- **appServiceSkuTier** `Standard`
 
 ## Functions Service Plan SKU
 
+**File Location:** `/infra/variables.tf`
 
-**File Location:** `/infra/main.bicep`
+### SKU Settings
 
-#### SKU Settings
-
-- **Name:** `S2`
-- **Capacity:** `2`
+- **functionsAppSkuSize** `S2`
+- **functionsAppSkuTie:** `Standard`
 
 ## Enrichment App Service Plan SKU
 
+**File Location:** `/infra/variables.tf`
 
-**File Location:** `/infra/main.bicep`
-
-#### SKU Settings
+### SKU Settings
 
-- **Name:** `P1v3`
-- **Tier:** `PremiumV3`
-- **Size:** `P1v3`
-- **Family:** `Pv3`
-- **Capacity:** `1`
+- **enrichmentAppServiceSkuSize** `P1v3`
+- **enrichmentAppServiceSkuTier** `PremiumV3`
 
 ### Enrichment Message Dequeue Parameter
-There exist a property that can be set in the local.env file called `DEQUEUE_MESSAGE_BATCH_SIZE` and is defaulted in the `infra/main.bicep` and `app/enrichment/app.py` to the value of **3**. This means the app will process 3 messages from the queue at a time. This is found to be the most optimal with the existing configuration but can be increased if you also increase the enrichment app service SKU. It is important to note that there will be issues if it is increased more than the app service SKU can handle.
+
+There exist a property called `DEQUEUE_MESSAGE_BATCH_SIZE` and is defaulted in the `infra/main.tf` to the value of **1**. This means the app will process 1 messages from the queue at a time. This is found to be the most optimal with the existing configuration but can be increased if you also increase the enrichment app service SKU. It is important to note that there will be issues if it is increased more than the app service SKU can handle.
 
 ### Customization
 
-To customize the App Service Plans SKU settings, modify the `sku` parameters in the specified Bicep file and run the `make deploy` or `make infrastructure`command.
+To customize the App Service Plans SKU settings, modify the terraform parameters by adding the following values to your `local.env` and run the `make deploy` or `make infrastructure`command.
+
+```bash
+export TF_VAR_functionsAppSkuSize="S2"
+export TF_VAR_functionsAppSkuTier="Standard"
+export TF_VAR_appServiceSkuSize="S1"
+export TF_VAR_appServiceSkuTier="Standard"
+export TF_VAR_enrichmentAppServiceSkuSize="P1V3"
+export TF_VAR_enrichmentAppServiceSkuTier="PremiumV3"
+```
 
 This can also be adjusted in the Azure Portal.
 
 **Note:** Adjusting the scale or Tier can cause outages until the redeployment occurs.
 
-
-### Steps to Scale Up:
+### Steps to Scale Up
 
 >1. **Sign in to the Azure Portal:**
 >   - Open a web browser and navigate to the [Azure Portal](https://portal.azure.com/).
 >   - Log in with your Azure account credentials.
-
 >2. **Navigate to the App Service:**
 >   - In the left navigation pane, select "App Services."
 >   - Click on the specific App Service you want to scale.
-
 >3. **Access the Scale Up Blade:**
 >   - In the App Service menu, find and click on "Scale up (App Service plan)" in the left sidebar.
-
 >4. **Choose a New Pricing Tier:**
 >   - On the "Scale Up" blade, you'll see different pricing tiers representing various levels of resources.
 >   - Select the desired pricing tier that corresponds to the scale you need.
-
 >5. **Review and Apply Changes:**
 >   - Review the information about the selected pricing tier, including its features and costs.
 >   - Click the "Apply" or "Save" button to apply the changes.
 
+### Considerations
 
-### Considerations:
 - **Cost Implications:**
   - Be aware of the cost implications associated with higher pricing tiers. Review the Azure Pricing documentation for details on costs.
 
@@ -146,4 +139,3 @@ This can also be adjusted in the Azure Portal.
 
 - **Performance Impact:**
   - Scaling up provides additional resources, potentially improving performance. However, it's essential to assess whether your application benefits from the increased resources.
-

docs/deployment/client_credentials_flow.md

@@ -4,7 +4,7 @@ If you want to call the backend API directly or want to use a custom User Interf
 
 > :warning: **Important Note**: Executing the instructions in this document will modify the authentication mechanism of the API to expect a token. This change will cause the out-of-the-box UI, included in the app service, to receive a 401 Unauthorized error on page load as it does not include the required token in its requests. Please ensure you have a strategy to handle this change in your UI before proceeding.  
 
-The below directions are showing how to accomplish via portal but most of which could be implemented in existing IaC through Bicep.
+The below directions are showing how to accomplish via portal but most of which could be implemented in existing IaC through Terraform.
 
 ## Update App Registration
 

docs/deployment/deployment.md

@@ -29,7 +29,7 @@ Once you have the completed setting up a GitHub Codespaces, please move on to th
 
  The IA Accelerator needs to be sized appropriately based on your use case. Please review our [Sizing Estimator](/docs/costestimator.md) to help find the configuration that fits your needs.
 
- To change the size of components deployed, make changes in the [Main Bicep](/infra/main.bicep) file.
+ To change the size of components deployed, make changes in the [Terraform Variables](/infra/variables.tf) file.
 
 Once you have completed the Sizing Estimator and sized your deployment appropriately, please move on to the Configuring your Environment step.
 
@@ -44,7 +44,7 @@ Inside your Development environment (GitHub Codespaces or Container), do the fol
 
 Variable | Required | Description
 --- | --- | ---
-LOCATION | Yes | The location (West Europe is the default). The BICEP templates use this value. To get a list of all the current Azure regions you can run `az account list-locations -o table`. The value here needs to be the *Name* value and not *Display Name*.
+LOCATION | Yes | The location (West Europe is the default). The Terraform templates use this value. To get a list of all the current Azure regions you can run `az account list-locations -o table`. The value here needs to be the *Name* value and not *Display Name*.
 WORKSPACE | Yes  | The workspace name (use something simple and unique to you). This will appended to infoasst-????? as the name of the resource group created in your subscription.
 SUBSCRIPTION_ID | Yes | The GUID that represents the Azure Subscription you want the Accelerator to be deployed into. This can be obtained from the *Subscription* blade in the Azure Portal.
 TENANT_ID | Yes | The GUID that represents the Azure Active Directory Tenant for the Subscription you want the accelerator to be deployed into. This can be obtained from the *Tenant Info* blade in the Azure Portal.
@@ -59,8 +59,8 @@ ENABLE_SHAREPOINT_CONNECTOR | Yes | Defaults to `false`. This feature flag enabl
 SHAREPOINT_TO_SYNC | No | This is a JSON Array of Objects for SharePoint Sites and their entry folders. The app will crawl down from the folder specified for each site. Specifying "/Shared Documents" will crawl all the documents in your SharePoint. `[{"url": "https://SharePoint.com/", "folder": "/Shared Documents"}]` This will **overwrite** any prior changes you've made to config.json. Information on setting up SharePoint Ingestion can be found here [SharePoint Connector](/docs/features/sharepoint.md)
 ENABLE_MULTIMEDIA | Yes | Defaults to `false`. This feature flag should not be changed at this time. The multimedia feature is still in development. Enabling this feature will deploy an Azure Video Indexer instance in your resource group only.
 REQUIRE_WEBSITE_SECURITY_MEMBERSHIP | Yes | Use this setting to determine whether a user needs to be granted explicit access to the website via an Azure AD Enterprise Application membership (true) or allow the website to be available to anyone in the Azure tenant (false). Defaults to false. If set to true, A tenant level administrator will be required to grant the implicit grant workflow for the Azure AD App Registration manually.
-SKIP_PLAN_CHECK | No | If this value is set to 1, then the BICEP deployment will not stop to allow you to review the planned changes. The default value is 0 in the scripts, which will allow the deployment to stop and confirm you accept the proposed changes before continuing.
-USE_EXISTING_AOAI | Yes | Defaults to false. Set this value to "true" if you want to use an existing Azure Open AI service instance in your subscription. This can be useful when there are limits to the number of AOAI instances you can have in one subscription. When the value is set to "false" and BICEP will create a new Azure Open AI service instance in your resource group.
+SKIP_PLAN_CHECK | No | If this value is set to 1, then the Terraform deployment will not stop to allow you to review the planned changes. The default value is 0 in the scripts, which will allow the deployment to stop and confirm you accept the proposed changes before continuing.
+USE_EXISTING_AOAI | Yes | Defaults to false. Set this value to "true" if you want to use an existing Azure Open AI service instance in your subscription. This can be useful when there are limits to the number of AOAI instances you can have in one subscription. When the value is set to "false" and Terraform will create a new Azure Open AI service instance in your resource group.
 AZURE_OPENAI_RESOURCE_GROUP | No | If you have set **USE_EXISTING_AOAI** to "true" then use this parameter to provide the name of the resource group that hosts the Azure Open AI service instance in your subscription.
 AZURE_OPENAI_SERVICE_NAME | No | If you have set **USE_EXISTING_AOAI** to "true" then use this parameter to provide the name of the Azure Open AI service instance in your subscription.
 AZURE_OPENAI_SERVICE_KEY | No | If you have set **USE_EXISTING_AOAI** to "true" then use this parameter to provide the Key for the Azure Open AI service instance in your subscription.
@@ -142,13 +142,13 @@ help                         Show this help
 deploy                       Deploy infrastructure and application code
 build                        Build application code
 infrastructure               Deploy infrastructure
-extract-env                  Extract infrastructure.env file from BICEP output
+extract-env                  Extract infrastructure.env file from Terraform output
 deploy-webapp                Deploys the web app code to Azure App Service
 deploy-functions             Deploys the function code to Azure Function Host
 deploy-enrichments           Deploys the web app code to Azure App Service
 deploy-search-indexes        Deploy search indexes
-extract-env-debug-webapp     Extract infrastructure.debug.env file from BICEP output
-extract-env-debug-functions  Extract local.settings.json to debug functions from BICEP output
+extract-env-debug-webapp     Extract infrastructure.debug.env file from Terraform output
+extract-env-debug-functions  Extract local.settings.json to debug functions from Terraform output
 functional-tests             Run functional tests to check the processing pipeline is working
 ```
 

docs/deployment/worbook_usage.md

@@ -45,6 +45,6 @@ To effectively use this template, follow these steps:
 
 ## Customization
 
-If you need to customize the default queries, locate the respective sections in the Bicep file and modify the `query` field within the `content` property.
+If you need to customize the default queries, locate the respective sections in the Terraform file and modify the `query` field within the `content` property.
 
 Feel free to adapt the template based on your specific log analysis requirements.

infra/README.md

@@ -26,7 +26,7 @@ build                           Build application code
 infrastructure                  Deploy infrastructure
 deploy-webapp                   Deploys the web app to Azure App Service
 deploy-search-indexes           Deploy search indexes
-extract-env                     Extract infrastructure.env file from BICEP output
+extract-env                     Extract infrastructure.env file from Terraform output
 ```
 
 ## Configure authentication and authorization

tests/README.md

@@ -4,7 +4,7 @@ The `/tests` folder contains a set of functional tests that validate the documen
 
 ## Functional tests
 
-The functional test are invoked as needed throughout the development process. It is initiated through a `make functional-tests` command which calls the `.\scripts\functional-tests.sh` script, which in turn parses the Bicep outputs and environment variables and invokes the Python-based functional tests. The goal of these is to make sure that throughout our development cycle, any changes made does not effect the expected processing pipeline outputs from a pre-determined set of input files (located in `.\tests`).
+The functional test are invoked as needed throughout the development process. It is initiated through a `make functional-tests` command which calls the `.\scripts\functional-tests.sh` script, which in turn parses the Terraform outputs and environment variables and invokes the Python-based functional tests. The goal of these is to make sure that throughout our development cycle, any changes made does not effect the expected processing pipeline outputs from a pre-determined set of input files (located in `.\tests`).
 
 To add more test cases, include new files for ingestions into the `.\tests\test_data` folder and name the file `test_example` with the filetype extension appropriate for the new test case.
 A search query for that file will need to be added to the test harness code near the top of the python file.