DESENG-811: Use Helm Charts in MET

.prettierignore

@@ -0,0 +1 @@
+*.yaml

CHANGELOG.MD

@@ -9,16 +9,80 @@
 - **Bugfix** Update SQLAlchemy library in MET ETL to match MET API [🎟️ DESENG-677](https://citz-gdx.atlassian.net/browse/DESENG-677)
 - **Bugfix** Add required dependencies for met-etl to requirements.txt [🎟️ DESENG-677](https://citz-gdx.atlassian.net/browse/DESENG-677)
   - pin flask_restx to 1.3.0
+- **Feature** Create Helm chart for analytics-api [🎟️ DESENG-811](https://citz-gdx.atlassian.net/browse/DESENG-811)
+  - Created a helm chart for the analytics-api deployment.
+  - Migrated existing Service and ConfigMap to Helm templates.
+  - Secrets now use HashiCorp Vault for sensitive data.
+  - Removed "debug=True" from all flask apps, as this is not appropriate for production.
+- **Feature** Update helm chart for MET Analytics (Redash) [🎟️ DESENG-811](https://citz-gdx.atlassian.net/browse/DESENG-811)
+  - Updated the helm chart for MET Analytics (Redash) to use Vault secrets for sensitive data.
+  - Removed Secrets file
+  - Removed sentiment analysis from the repository, as it is no longer used.
+  - Change from Ingress to Route for consistency with other applications.
+  - Properly randomized the secret keys for redash and the database
 
 ## May 20, 2025
 
 - **Bugfix** Remove potentially redundant Dagster libraries [🎟️ DESENG-677](https://citz-gdx.atlassian.net/browse/DESENG-677)
 - Remove pip `-U` flag from installation commands to prevent version instabilities
 
+## May 14, 2025
+
+- **Feature** Create Helm chart for notify-api [🎟️ DESENG-811](https://citz-gdx.atlassian.net/browse/DESENG-811)
+  - Created a helm chart for the notify-api deployment.
+  - Migrated existing Service and ConfigMap to Helm templates.
+  - Removed notify-api Route, as the notify-api does not need to be publicly accessible. Instead, it is called by the MET API & MET Cron jobs.
+    - For local development, the notify-api can be accessed via port-forwarding. See DEVELOPMENT.md for more information.
+  - Added new "global" Vault secret for all flask apps to use, which contains the `SECRET_KEY`.
+  - Migrated the DeploymentConfig to a Helm template.
+  - Created values files (dev, test, prod) for deployment settings; see `openshift/README.md`
+
 ## May 13, 2025
 
 - **Bugfix** Align library versions with EPIC app [🎟️ DESENG-677](https://citz-gdx.atlassian.net/browse/DESENG-677)
 
+## May 12, 2025
+
+- **Feature** Add MET Cron job to OpenShift [🎟️ DESENG-811](https://citz-gdx.atlassian.net/browse/DESENG-811)
+  - Added the met-cron deployment to the met-api helm chart.
+  - Removed an unneeded Service, as the Cron does not serve any traffic.
+  - Set up injection of HashiCorp Vault secrets to the Cron job
+  - Merged met-cron ConfigMap into the met-api ConfigMap, as they differed very little.
+  - Renamed config key "OFFSET_DAYS" to "CLOSING_SOON_EMAIL_ADVANCE_NOTICE_DAYS" to be more descriptive.
+  - Removed the old met-cron deployment YAML files.
+
+## May 7, 2025
+
+- **Feature** Add Helm chart and templates for MET Web [🎟️ DESENG-811](https://citz-gdx.atlassian.net/browse/DESENG-811)
+  - Created a helm chart for the MET Web deployment.
+  - Migrated existing resources to Helm templates.
+  - Removed several unused ConfigMap values
+  - Removed JWT/OIDC related values in favor of requesting the data from the MET API (single source of truth).
+    - Created an endpoint at `/api/oidc_config` to return the OIDC configuration for the MET API.
+  - Modified the Docker entrypoint to generate a config script from REACT_APP\_\* environment variables at startup.
+    - The config script is then served alongside the app at runtime.
+- **Bugfix** Updated existing templates to set minReplicas and maxReplicas for the HPA (matching met-web) rather than trying to configure it on the Deployment, which has no effect.
+
+## May 5, 2025
+
+- **Feature** Add Helm chart and templates for MET API [🎟️ DESENG-811](https://citz-gdx.atlassian.net/browse/DESENG-811)
+  - Created a helm chart for the MET API deployment.
+  - Migrated existing HorizontalPodAutoscaler, Route, Service, and MET ConfigMap to Helm templates.
+  - Removed Secrets in favor of using HashiCorp Vault for sensitive data.
+    - Also removed several ConfigMap values that were either sensitive or were contextually linked to
+      the secrets, and moved their data to the Vault.
+    - Developers can now use `helm` to generate the Openshift YAML files without needing
+      to seek out secrets and keep them within their values files.
+  - Migrated the DeploymentConfig to a Helm template.
+    - Modified heavily to use new Vault secrets. Secrets are now mounted as files in the
+      container at /vault/secrets, and the application is expected to read them from the filesystem.
+    - Bound the DeploymentConfig to the e903c2-vault service account, which has access to the
+      secrets in the vault. Also granted the vault account image pull permissions from e903c2-tools.
+  - Created environment-specific values files (dev, test, prod) for deployment settings.
+    - See `openshift/README.md` for more information on how to deploy the API using the new Helm chart.
+  - Removed YAML for the old Keycloak server, since we have migrated to using the shared SSO Keycloak
+    server.
+
 ## May 1, 2025
 
 - **Bugfix** Resolve conflicting versions for Dagster ETL pipeline [🎟️ DESENG-677](https://citz-gdx.atlassian.net/browse/DESENG-677)

DEVELOPMENT.md

@@ -34,7 +34,9 @@ npm run test
 ```
 
 ## met-api
+
 ## analytics-api
+
 ## notify-api
 
 Create a .env file based on the sample.env
@@ -46,7 +48,7 @@ make setup
 ```
 
 Manually upgrading the database:
-*This wil also create some default data for the app if it does not exist*
+_This will also create some default data for the app if it does not exist_
 
 ```
 make db
@@ -64,6 +66,15 @@ Running the unit test:
 make test
 ```
 
+Create a port forward to the email service hosted on OpenShift:
+
+```
+cd ./notify-api
+make port-forward
+# Or, if you prefer not to use make:
+oc port-forward svc/notify-api 8081:8080 -n e903c2-dev
+```
+
 ## met-cron
 
 Create a .env file based on the sample.env
@@ -79,6 +90,7 @@ This is a task scheduler project, to run tasks manually use the following comman
 ```
 make run_closeout
 ```
+
 ```
 make run_publish
 ```
@@ -98,16 +110,19 @@ docker compose up
 A custom redash project is used for some of the dashboards whithin MET.
 
 To start an instance clone the following repository:
+
 ```
 git clone https://github.com/bcgov/redash
 ```
 
 create a .env file with the following:
+
 ```
 REDASH_COOKIE_SECRET=redash
 ```
 
 Run the docker compose command:
+
 ```
 docker compose up
-```
+```

analytics-api/Makefile

@@ -46,20 +46,20 @@ build-req: clean ## Upgrade requirements
 	test -f venv/bin/activate || python3.12 -m venv  $(CURRENT_ABS_DIR)/venv ;\
 	. venv/bin/activate ;\
 	pip install --upgrade pip ;\
-	pip install -Ur requirements/prod.txt ;\
+	pip install -r requirements/prod.txt ;\
 	pip freeze | sort > requirements.txt ;\
 	cat requirements/repo-libraries.txt >> requirements.txt ;\
-	pip install -Ur requirements/repo-libraries.txt
+	pip install -r requirements/repo-libraries.txt
 
 install: clean ## Install python virtual environment
 	test -f venv/bin/activate || python3 -m venv  $(CURRENT_ABS_DIR)/venv ;\
 	. venv/bin/activate ;\
 	pip install --upgrade pip ;\
-	pip install -Ur requirements.txt
+	pip install -r requirements.txt
 
 install-dev: ## Install local application
 	. venv/bin/activate ; \
-	pip install -Ur requirements/dev.txt; \
+	pip install -r requirements/dev.txt; \
 	pip install -e .
 
 #################################################################################

analytics-api/requirements/prod.txt

@@ -27,7 +27,7 @@ Flask-Caching
 asyncio-nats-client
 asyncio-nats-streaming
 sqlalchemy==1.4.52
-secure
+secure==0.3.0
 python-dotenv
 aws-requests-auth
 requests

analytics-api/wsgi.py

@@ -3,4 +3,5 @@
 application = create_app()
 
 if __name__ == "__main__":
-    application.run(debug=True, host='0.0.0.0', port=5001)
+    # Never set debug=True in production
+    application.run(debug=False, host='0.0.0.0', port=5001)

met-api/sample.env

@@ -67,7 +67,7 @@ SQLALCHEMY_ECHO=
 SQLALCHEMY_TRACK_MODIFICATIONS=
 
 # Email API Configuration
-NOTIFICATIONS_EMAIL_ENDPOINT=https://met-notify-api-dev.apps.gold.devops.gov.bc.ca/api/v1/notifications/email
+NOTIFICATIONS_EMAIL_ENDPOINT=http://localhost:8081/api/v1/notifications/email
 EMAIL_SECRET_KEY="notASecureKey" # If unset, this value is randomized
 EMAIL_ENVIRONMENT=
 EMAIL_FROM_ADDRESS="[email protected]"

met-api/src/met_api/config.py

@@ -271,6 +271,7 @@ def SQLALCHEMY_DATABASE_URI(self) -> str:
         # The time of day when engagements get closed. This should match the
         # value in met-cron/cron/crontab
         'CLOSING_TIME': os.getenv('ENGAGEMENT_END_TIME', '5 PM'),
+        # This only applies when using GC Notify instead of CHES
         'FROM_ADDRESS': os.getenv('EMAIL_FROM_ADDRESS'),
         'ENVIRONMENT': os.getenv('EMAIL_ENVIRONMENT'),
         'SUBSCRIBE': {

met-api/src/met_api/resources/__init__.py

@@ -25,6 +25,7 @@
 from flask import Blueprint
 
 from .apihelper import Api
+from .oidc_config import API as OIDC_CONFIG_API
 from .comment import API as COMMENT_API
 from .contact import API as CONTACT_API
 from .document import API as DOCUMENT_API
@@ -77,6 +78,7 @@
 
 # HANDLER = ExceptionHandler(API)
 
+API.add_namespace(OIDC_CONFIG_API, path='/oidc_config')
 API.add_namespace(ENGAGEMENT_API)
 API.add_namespace(USER_API)
 API.add_namespace(DOCUMENT_API)

met-api/src/met_api/resources/oidc_config.py

@@ -0,0 +1,57 @@
+"""A simple endpoint to serve the OpenID Connect configuration for the web application."""
+
+from flask import jsonify
+from flask_cors import cross_origin
+from flask_restx import Namespace, Resource
+from met_api.utils.roles import Role
+from met_api.utils.util import allowedorigins, cors_preflight
+from met_api.config import Config
+
+API = Namespace(
+    'oidc_config',
+    description='Endpoints for fetching OpenID Connect configuration',
+)
+
+jwt_config = Config().JWT
+keycloak_config = Config().KC
+
+PUBLIC_CONFIG = {
+    # Do not overpopulate this dict with sensitive information
+    # as it will be intentionally exposed to the public
+    'KEYCLOAK_URL': keycloak_config['BASE_URL'],
+    'KEYCLOAK_REALM': keycloak_config['REALMNAME'],
+    'KEYCLOAK_CLIENT': jwt_config['AUDIENCE'],
+    'KEYCLOAK_ADMIN_ROLE': Role.SUPER_ADMIN.value,
+}
+
+
+@cors_preflight('GET,OPTIONS')
+@API.route('/')
+class OIDCConfigAsJson(Resource):
+    """Resource for OpenID Connect configuration."""
+
+    @staticmethod
+    @cross_origin(origins=allowedorigins())
+    def get():
+        """Fetch OpenID Connect configuration."""
+        return jsonify(PUBLIC_CONFIG), 200
+
+
+@cors_preflight('GET,OPTIONS')
+@API.route('/config.js')
+class OIDCConfigAsJs(Resource):
+    """Resource for OpenID Connect configuration in JavaScript format."""
+
+    js_prefix = 'window._env_ = window._env_ || {};\n'
+    js_template = "window._env_.{VAR_NAME} = '{VAR_VALUE}';\n"
+
+    @staticmethod
+    @cross_origin(origins=allowedorigins())
+    def get():
+        """Fetch OpenID Connect configuration."""
+        js_content = OIDCConfigAsJs.js_prefix
+        for key, value in PUBLIC_CONFIG.items():
+            js_content += OIDCConfigAsJs.js_template.format(
+                VAR_NAME='REACT_APP_' + key, VAR_VALUE=value
+            )
+        return js_content, 200, {'Content-Type': 'application/javascript'}

met-api/src/met_api/schemas/widget_listening.py

@@ -16,7 +16,7 @@
 from met_api.models.widget_listening import WidgetListening as WidgetListeningModel
 
 from marshmallow import Schema
-from marshmallow_sqlalchemy.fields import Nested
+
 
 class WidgetListeningSchema(Schema):  # pylint: disable=too-many-ancestors, too-few-public-methods
     """This is the schema for the widget listening model."""

met-api/src/met_api/services/widget_translation_service.py

@@ -132,7 +132,7 @@ def _get_default_language_values(widget, translation_data):
             if widget_video:
                 translation_data['video_url'] = widget_video[0].video_url
                 translation_data['video_description'] = widget_video[0].description
-                
+
         if widget_type == WidgetType.WHO_IS_LISTENING.value:
             widget_listening = WidgetListeningModel.get_listening(widget_id)
             if widget_listening: