Merge branch 'develop' into ocrvs-10672

Author: adskyiproger

README.md

@@ -23,20 +23,35 @@
 
 This is an example country configuration package for the OpenCRVS core. OpenCRVS requires a country configuration in order to run.
 
-OpenCRVS is designed to be highly configurable for your country needs. It achieves this by seeding reference data that it needs from this module and exposing APIs for certain business critical operations. 
+OpenCRVS is designed to be highly configurable for your country needs. It achieves this by seeding reference data that it needs from this module and exposing APIs for certain business critical operations.
 
 This module also provides a logical location where you may wish to store the code and run the servers for any custom API integrations, extension modules and innovations to OpenCRVS.
 
 ## How do I run the module alongside the OpenCRVS core?
 
-1. Ensure that you are running [OpenCRVS Core](https://github.com/opencrvs/opencrvs-core).  
+1. Ensure that you are running [OpenCRVS Core](https://github.com/opencrvs/opencrvs-core).
 
-**If you successfully ran the `bash setup.sh` script in OpenCRVS Core you already have this module checked out, the dependencies are installed, the Farajaland database is populated and you can just run the following command.** 
+**If you successfully ran the `bash setup.sh` script in OpenCRVS Core you already have this module checked out, the dependencies are installed, the Farajaland database is populated and you can just run the following command.**
 
 2. `yarn dev`
 
 Thats it! 🎉
 
+## Database Management
+
+The country configuration includes scripts for managing analytics and metabase databases during development and deployment.
+
+### Development Database Management
+
+- `yarn db:clear:all` - Clears all development databases including:
+  - PostgreSQL analytics schema and data
+The script is designed to be extended by country implementations to clear additional custom databases as needed.
+
+### Production Database Management
+
+For deployed environments, analytics data is cleared when the "reset data" pipeline is triggered, which uses the `infrastructure/clear-all-data.sh` script. This script now includes clearing of:
+- **Metabase analytics database** - Both the PostgreSQL analytics schema and Metabase H2 configuration database
+
 ## What is in the Farajaland configuration module repository?
 
 One of the key dependencies and enablers for OpenCRVS is country configuration and a reference data source. This source is bespoke for every implementing nation. If you would like to create your own country implementation, we recommend that you duplicate this repository and use it as a template. So what does it contain?
@@ -57,7 +72,7 @@ When the OpenCRVS Core servers start up with un-seeded databases they call the f
 
 1. `GET /application-config`
 
-- Configures general application settings 
+- Configures general application settings
 
 2. `GET /users`
 

infrastructure/backups/restore.sh

@@ -149,8 +149,9 @@ echo "🧹 cleanup for indices: $approved_words from $indices"
 echo "--------------------------"
 for index in ${indices[@]}; do
   for approved in $approved_words; do
+    echo "Checking index $index against approved pattern $approved..."
     case "$index" in
-    "$approved_words"*) 
+    "$approved"*)
         echo "Delete index $index..."
         docker run --rm --network=$NETWORK appropriate/curl curl -sS -XDELETE "http://$(elasticsearch_host)/$index"
         break
@@ -214,7 +215,7 @@ if [ -f "$ROOT_PATH/backups/postgres/events-${LABEL}.dump" ]; then
     -e PGPASSWORD=$POSTGRES_PASSWORD \
     --network=$NETWORK \
     postgres:17.6 \
-    bash -c "psql -h postgres -U $POSTGRES_USER -c 'DROP DATABASE IF EXISTS events;'"
+    bash -c "psql -h postgres -U $POSTGRES_USER -c 'DROP DATABASE IF EXISTS events WITH (FORCE);'"
 else
   echo "PostgreSQL backup not found for label ${LABEL}. Skipping PostgreSQL database drop..."
 fi
@@ -252,7 +253,11 @@ if [ -f "$ROOT_PATH/backups/postgres/events-${LABEL}.dump" ]; then
     -v $ROOT_PATH/backups/postgres:/backups \
     --network=$NETWORK \
     postgres:17.6 \
-    bash -c "createdb -h postgres -U $POSTGRES_USER events && pg_restore -h postgres -U $POSTGRES_USER -d events /backups/events-${LABEL}.dump"
+    bash -c "createdb -h postgres -U $POSTGRES_USER events && \
+             psql -h postgres -U $POSTGRES_USER -d events -c 'CREATE SCHEMA app AUTHORIZATION events_migrator; GRANT USAGE ON SCHEMA app TO events_app;' && \
+             pg_restore -h postgres -U $POSTGRES_USER -d events --schema=app /backups/events-${LABEL}.dump"
+  echo "Update credentials in Postgres on restore"
+  docker service update --force opencrvs_postgres-on-update
 else
   echo "PostgreSQL backup not found for label ${LABEL}. Skipping PostgreSQL database restore..."
 fi
@@ -309,3 +314,15 @@ tar -xzvf $ROOT_PATH/backups/vsexport/ocrvs-$LABEL.tar.gz -C $ROOT_PATH/vsexport
 if [ "$IS_LOCAL" = false ]; then
   docker service update --force --update-parallelism 1 opencrvs_migration
 fi
+
+##
+# ------ REINDEX -----
+##
+docker run --rm \
+  -v /opt/opencrvs/infrastructure/deployment:/workspace \
+  -w /workspace \
+  --network $NETWORK \
+  -e 'AUTH_URL=http://auth:4040/' \
+  -e 'EVENTS_URL=http://gateway:7070/events' \
+  alpine \
+  sh -c 'apk add --no-cache curl jq && sh reindex.sh'

infrastructure/clear-all-data-dev.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# OpenCRVS is also distributed under the terms of the Civil Registration
+# & Healthcare Disclaimer located at http://opencrvs.org/license.
+#
+# Copyright (C) The OpenCRVS Authors located at https://github.com/opencrvs/opencrvs-core/blob/master/AUTHORS.
+
+set -e
+
+echo "🧹 Clearing all development data..."
+
+# Default PostgreSQL connection parameters for development
+POSTGRES_HOST=${POSTGRES_HOST:-localhost}
+POSTGRES_PORT=${POSTGRES_PORT:-5432}
+POSTGRES_USER=${POSTGRES_USER:-postgres}
+POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-postgres}
+ANALYTICS_POSTGRES_USER=${ANALYTICS_POSTGRES_USER:-events_analytics}
+
+print_usage_and_exit() {
+  echo 'Usage: ./clear-all-data-dev.sh'
+  echo ""
+  echo "Environment variables (with defaults for development):"
+  echo "POSTGRES_HOST=${POSTGRES_HOST}"
+  echo "POSTGRES_PORT=${POSTGRES_PORT}"
+  echo "POSTGRES_USER=${POSTGRES_USER}"
+  echo "POSTGRES_PASSWORD=${POSTGRES_PASSWORD}"
+  echo "ANALYTICS_POSTGRES_USER=${ANALYTICS_POSTGRES_USER}"
+  echo ""
+  echo "This script clears all development databases including:"
+  echo "- PostgreSQL analytics schema and data"
+  echo "- Metabase configuration database (H2)"
+  echo "- Any additional custom databases (can be extended by country configurations)"
+  exit 1
+}
+
+# Check if PostgreSQL is accessible
+if ! PGPASSWORD="$POSTGRES_PASSWORD" psql -h "$POSTGRES_HOST" -p "$POSTGRES_PORT" -U "$POSTGRES_USER" -d postgres -c '\q' 2>/dev/null; then
+  echo "❌ Cannot connect to PostgreSQL at ${POSTGRES_HOST}:${POSTGRES_PORT}"
+  echo "   Make sure PostgreSQL is running and credentials are correct."
+  print_usage_and_exit
+fi
+
+# Clear PostgreSQL analytics schema
+echo "🗑️  Clearing PostgreSQL analytics schema..."
+PGPASSWORD="$POSTGRES_PASSWORD" psql -h "$POSTGRES_HOST" -p "$POSTGRES_PORT" -U "$POSTGRES_USER" -d events -v ON_ERROR_STOP=1 <<EOSQL || echo "⚠️  Analytics schema may not exist yet"
+-- Drop analytics schema and recreate it
+DROP SCHEMA IF EXISTS analytics CASCADE;
+CREATE SCHEMA analytics;
+
+-- Drop and recreate analytics user
+DROP ROLE IF EXISTS "$ANALYTICS_POSTGRES_USER";
+EOSQL
+
+echo "✅ PostgreSQL analytics data cleared"
+
+echo ""
+echo "🎉 All development data cleared successfully!"
+echo ""

infrastructure/clear-all-data.sh

@@ -127,8 +127,10 @@ docker run --rm --network=$NETWORK --entrypoint=/bin/sh minio/mc:RELEASE.2025-05
 #-------------------------------
 
 POSTGRES_DB="events"
+ANALYTICS_POSTGRES_DB="analytics"
 EVENTS_MIGRATOR_ROLE="events_migrator"
 EVENTS_APP_ROLE="events_app"
+ANALYTICS_POSTGRES_ROLE=${ANALYTICS_POSTGRES_USER:-"events_analytics"}
 
 echo "🔁 Dropping database '${POSTGRES_DB}' and roles..."
 
@@ -138,17 +140,21 @@ docker run --rm --network=$NETWORK  \
   -e POSTGRES_DB="${POSTGRES_DB}" \
   -e EVENTS_MIGRATOR_ROLE="${EVENTS_MIGRATOR_ROLE}" \
   -e EVENTS_APP_ROLE="${EVENTS_APP_ROLE}" \
+  -e ANALYTICS_POSTGRES_ROLE="${ANALYTICS_POSTGRES_ROLE}" \
+  -e ANALYTICS_POSTGRES_DB="${ANALYTICS_POSTGRES_DB}" \
   postgres:17.6 bash -c '
 psql -h postgres -U "$POSTGRES_USER" -d postgres -v ON_ERROR_STOP=1 <<EOF
 DROP DATABASE IF EXISTS "$POSTGRES_DB" WITH (FORCE);
+DROP DATABASE IF EXISTS "$ANALYTICS_POSTGRES_DB" WITH (FORCE);
 
 DROP ROLE IF EXISTS "$EVENTS_MIGRATOR_ROLE";
 DROP ROLE IF EXISTS "$EVENTS_APP_ROLE";
+DROP ROLE IF EXISTS "$ANALYTICS_POSTGRES_ROLE";
 EOF
 '
 echo "✅ Database and roles dropped."
 echo "🚀 Reinitializing Postgres with on-deploy.sh..."
 
 docker service update --force opencrvs_postgres-on-update
 
-echo "✅ All data cleared."
+echo "✅ All data cleared."

infrastructure/metabase/README.md

@@ -10,12 +10,20 @@
 
 # OpenCRVS Dashboards
 
+### Requirements
+
+```
+❯ java -version
+openjdk version "21.0.8" 2025-07-15
+```
+
 ### Run in development mode
 
 By default, Metabase is not started as part of the OpenCRVS stack as running it requires quite a bit of resources. You can use the following commands to use metabase in development:
 
 `yarn metabase` – Starts Metabase in port http://localhost:4444
 `yarn db:shell` - In the metabase directory – open a SQL shell for the database Metabase created
+`yarn db:clear:all` - Clear data from analytics dashboard
 
 ### Default credentials