Merge pull request #1041 from opencrvs/ocrvs-10559

Add development time yarn db:clear:all script for easy analytics data clearing

Author: rikukissa

README.md

@@ -23,20 +23,35 @@
 
 This is an example country configuration package for the OpenCRVS core. OpenCRVS requires a country configuration in order to run.
 
-OpenCRVS is designed to be highly configurable for your country needs. It achieves this by seeding reference data that it needs from this module and exposing APIs for certain business critical operations. 
+OpenCRVS is designed to be highly configurable for your country needs. It achieves this by seeding reference data that it needs from this module and exposing APIs for certain business critical operations.
 
 This module also provides a logical location where you may wish to store the code and run the servers for any custom API integrations, extension modules and innovations to OpenCRVS.
 
 ## How do I run the module alongside the OpenCRVS core?
 
-1. Ensure that you are running [OpenCRVS Core](https://github.com/opencrvs/opencrvs-core).  
+1. Ensure that you are running [OpenCRVS Core](https://github.com/opencrvs/opencrvs-core).
 
-**If you successfully ran the `bash setup.sh` script in OpenCRVS Core you already have this module checked out, the dependencies are installed, the Farajaland database is populated and you can just run the following command.** 
+**If you successfully ran the `bash setup.sh` script in OpenCRVS Core you already have this module checked out, the dependencies are installed, the Farajaland database is populated and you can just run the following command.**
 
 2. `yarn dev`
 
 Thats it! 🎉
 
+## Database Management
+
+The country configuration includes scripts for managing analytics and metabase databases during development and deployment.
+
+### Development Database Management
+
+- `yarn db:clear:all` - Clears all development databases including:
+  - PostgreSQL analytics schema and data
+The script is designed to be extended by country implementations to clear additional custom databases as needed.
+
+### Production Database Management
+
+For deployed environments, analytics data is cleared when the "reset data" pipeline is triggered, which uses the `infrastructure/clear-all-data.sh` script. This script now includes clearing of:
+- **Metabase analytics database** - Both the PostgreSQL analytics schema and Metabase H2 configuration database
+
 ## What is in the Farajaland configuration module repository?
 
 One of the key dependencies and enablers for OpenCRVS is country configuration and a reference data source. This source is bespoke for every implementing nation. If you would like to create your own country implementation, we recommend that you duplicate this repository and use it as a template. So what does it contain?
@@ -57,7 +72,7 @@ When the OpenCRVS Core servers start up with un-seeded databases they call the f
 
 1. `GET /application-config`
 
-- Configures general application settings 
+- Configures general application settings
 
 2. `GET /users`
 

infrastructure/clear-all-data-dev.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# OpenCRVS is also distributed under the terms of the Civil Registration
+# & Healthcare Disclaimer located at http://opencrvs.org/license.
+#
+# Copyright (C) The OpenCRVS Authors located at https://github.com/opencrvs/opencrvs-core/blob/master/AUTHORS.
+
+set -e
+
+echo "🧹 Clearing all development data..."
+
+# Default PostgreSQL connection parameters for development
+POSTGRES_HOST=${POSTGRES_HOST:-localhost}
+POSTGRES_PORT=${POSTGRES_PORT:-5432}
+POSTGRES_USER=${POSTGRES_USER:-postgres}
+POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-postgres}
+ANALYTICS_POSTGRES_USER=${ANALYTICS_POSTGRES_USER:-events_analytics}
+
+print_usage_and_exit() {
+  echo 'Usage: ./clear-all-data-dev.sh'
+  echo ""
+  echo "Environment variables (with defaults for development):"
+  echo "POSTGRES_HOST=${POSTGRES_HOST}"
+  echo "POSTGRES_PORT=${POSTGRES_PORT}"
+  echo "POSTGRES_USER=${POSTGRES_USER}"
+  echo "POSTGRES_PASSWORD=${POSTGRES_PASSWORD}"
+  echo "ANALYTICS_POSTGRES_USER=${ANALYTICS_POSTGRES_USER}"
+  echo ""
+  echo "This script clears all development databases including:"
+  echo "- PostgreSQL analytics schema and data"
+  echo "- Metabase configuration database (H2)"
+  echo "- Any additional custom databases (can be extended by country configurations)"
+  exit 1
+}
+
+# Check if PostgreSQL is accessible
+if ! PGPASSWORD="$POSTGRES_PASSWORD" psql -h "$POSTGRES_HOST" -p "$POSTGRES_PORT" -U "$POSTGRES_USER" -d postgres -c '\q' 2>/dev/null; then
+  echo "❌ Cannot connect to PostgreSQL at ${POSTGRES_HOST}:${POSTGRES_PORT}"
+  echo "   Make sure PostgreSQL is running and credentials are correct."
+  print_usage_and_exit
+fi
+
+# Clear PostgreSQL analytics schema
+echo "🗑️  Clearing PostgreSQL analytics schema..."
+PGPASSWORD="$POSTGRES_PASSWORD" psql -h "$POSTGRES_HOST" -p "$POSTGRES_PORT" -U "$POSTGRES_USER" -d events -v ON_ERROR_STOP=1 <<EOSQL || echo "⚠️  Analytics schema may not exist yet"
+-- Drop analytics schema and recreate it
+DROP SCHEMA IF EXISTS analytics CASCADE;
+CREATE SCHEMA analytics;
+
+-- Drop and recreate analytics user
+DROP ROLE IF EXISTS "$ANALYTICS_POSTGRES_USER";
+EOSQL
+
+echo "✅ PostgreSQL analytics data cleared"
+
+echo ""
+echo "🎉 All development data cleared successfully!"
+echo ""

infrastructure/clear-all-data.sh

@@ -127,8 +127,10 @@ docker run --rm --network=$NETWORK --entrypoint=/bin/sh minio/mc:RELEASE.2025-05
 #-------------------------------
 
 POSTGRES_DB="events"
+ANALYTICS_POSTGRES_DB="analytics"
 EVENTS_MIGRATOR_ROLE="events_migrator"
 EVENTS_APP_ROLE="events_app"
+ANALYTICS_POSTGRES_ROLE=${ANALYTICS_POSTGRES_USER:-"events_analytics"}
 
 echo "🔁 Dropping database '${POSTGRES_DB}' and roles..."
 
@@ -138,17 +140,28 @@ docker run --rm --network=$NETWORK  \
   -e POSTGRES_DB="${POSTGRES_DB}" \
   -e EVENTS_MIGRATOR_ROLE="${EVENTS_MIGRATOR_ROLE}" \
   -e EVENTS_APP_ROLE="${EVENTS_APP_ROLE}" \
+  -e ANALYTICS_POSTGRES_USER="${ANALYTICS_POSTGRES_USER}" \
   postgres:17.6 bash -c '
 psql -h postgres -U "$POSTGRES_USER" -d postgres -v ON_ERROR_STOP=1 <<EOF
 DROP DATABASE IF EXISTS "$POSTGRES_DB" WITH (FORCE);
+DROP DATABASE IF EXISTS "$ANALYTICS_POSTGRES_DB" WITH (FORCE);
 
 DROP ROLE IF EXISTS "$EVENTS_MIGRATOR_ROLE";
 DROP ROLE IF EXISTS "$EVENTS_APP_ROLE";
+DROP ROLE IF EXISTS "$ANALYTICS_POSTGRES_ROLE";
 EOF
 '
 echo "✅ Database and roles dropped."
 echo "🚀 Reinitializing Postgres with on-deploy.sh..."
 
 docker service update --force opencrvs_postgres-on-update
 
-echo "✅ All data cleared."
+docker service scale opencrvs_dashboards=0
+docker service scale opencrvs_dashboards=1
+
+# Restart events service
+#-----------------------------
+docker service scale opencrvs_events=0
+docker service scale opencrvs_events=1
+
+echo "✅ All data cleared."

package.json

@@ -22,6 +22,7 @@
     "test:compilation": "tsc --noEmit",
     "lint": "eslint -c .eslintrc.js",
     "setup-analytics": "docker exec -i -e POSTGRES_HOST=localhost -e POSTGRES_PORT=5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e ANALYTICS_POSTGRES_USER=events_analytics -e ANALYTICS_POSTGRES_PASSWORD=analytics_password postgres bash -s < ./infrastructure/postgres/setup-analytics.sh",
+    "db:clear:all": "bash infrastructure/clear-all-data-dev.sh && yarn setup-analytics",
     "start": "yarn setup-analytics && cross-env NODE_ENV=development NODE_OPTIONS=--dns-result-order=ipv4first nodemon --exec ts-node -r tsconfig-paths/register src/index.ts",
     "start:prod": "ts-node --transpile-only -r tsconfig-paths/register src/index.ts",
     "metabase": "bash infrastructure/metabase/run-dev.sh",