opencrvs · jamil314 · Nov 6, 2025 · Oct 6, 2025 · Oct 6, 2025 · Oct 6, 2025
diff --git a/ANALYTICS.md b/ANALYTICS.md
@@ -0,0 +1,182 @@
+# OpenCRVS Analytics Documentation
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Overview](#overview)
+- [Architecture](#architecture)
+- [Data Sources](#data-sources)
+- [How Data Flows to Metabase](#how-data-flows-to-metabase)
+- [Available Data in Example Setup](#available-data-in-example-setup)
+- [Deployment](#deployment)
+- [Development and Dashboard Management](#development-and-dashboard-management)
+- [Why Analytics Data is Not Backed Up](#why-analytics-data-is-not-backed-up)
+- [Configuration](#configuration)
+- [Development Commands](#development-commands)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Overview
+
+OpenCRVS provides comprehensive analytics capabilities through **Metabase**, an open-source business intelligence platform. This system allows stakeholders to visualize and analyze civil registration data through interactive dashboards, charts, and reports.
+
+The analytics system is designed to:
+- Track vital events (births, deaths, etc.) and registration statistics
+- Provide insights for decision-making and reporting
+- Support data-driven improvements to civil registration processes
+- Export non-PII datasets as CSV
+
+## Architecture
+
+The analytics system consists of several key components:
+
+```
+┌──────────────────┐    ┌──────────────────┐    ┌─────────────────┐
+│  OpenCRVS Core   │───▶│   Country config │───▶│    Metabase     │
+│  Action trigger  │    │   Analytics DB   │    │   Dashboards    │
+└──────────────────┘    └──────────────────┘    └─────────────────┘
+```
+
+1. **OpenCRVS Core Services**: Submits all performed event actions to country config
+2. **PostgreSQL Analytics Database**: Stores processed analytics data in country-defined database. By default this is in PostgreSQL in `analytics` schema
+3. **Metabase**: Connects to PostgreSQL and provides visualization capabilities
+
+## Data Sources
+
+OpenCRVS collects analytics data from multiple sources:
+
+### Event Data
+- **Birth registrations**: Demographics, locations, registration timing
+- **Death registrations**: Cause of death, demographics, locations
+- **Custom events**: Additional event types (e.g., tennis club memberships in the example)
+- **Registration status**: Draft, registered, certified states
+- **Action history**: Workflow actions and state transitions
+
+Country config receives full event documents and can decide which fields should be written into the analytics database and which are PII.
+
+### Configuration-Driven Analytics
+The example country config implementation uses configurable analytics fields marked with `analytics: true` in form configurations:
+
+```typescript
+// Only fields marked with analytics: true are included
+field: {
+  id: 'childDetails.firstName',
+  analytics: true,  // This field will be tracked
+  // ... other field properties
+}
+```
+
+## How Data Flows to Metabase
+
+1. **Event Processing**: When an event receives an action in OpenCRVS (registration, status changes, etc.), country config HTTP action hooks are triggered and the system processes the action through the analytics pipeline defined in `src/analytics/analytics.ts`
+
+2. **Data Extraction**: The analytics service extracts relevant fields from event documents based on form configuration (only fields marked with `analytics: true`)
+
+3. **Data Transformation**: Raw event data is transformed and enriched with additional calculated metrics (e.g., registration delays, demographic statistics).
+
+4. **Database Storage**: Processed data is stored in PostgreSQL in the `analytics` schema, specifically in the `event_actions` table
+
+5. **Metabase Connection**: Metabase connects directly to the PostgreSQL database using configured credentials and queries the analytics schema
+
+6. **Dashboard Rendering**: Pre-configured dashboards and queries in Metabase visualize the data through charts, tables, and maps
+
+## Deployment
+
+### Production Deployment
+Metabase is deployed as a Docker service defined in `infrastructure/docker-compose.deploy.yml`:
+
+```yaml
+dashboards:
+  image: metabase/metabase:v0.56.4
+  volumes:
+    - /opt/opencrvs/infrastructure/metabase/metabase.init.db.sql:/metabase.init.db.sql
+    - /opt/opencrvs/infrastructure/metabase/run.sh:/run.sh
+    # ... other configuration files
+  environment:
+    # Database connection settings
+    - METABASE_DATABASE_HOST=${METABASE_DATABASE_HOST:-postgres}
+    - METABASE_DATABASE_NAME=${METABASE_DATABASE_NAME:-events}
+    - METABASE_DATABASE_USER=${METABASE_DATABASE_USER:-events_analytics}
+    # ... other environment variables
+```
+
+### Environment Configuration
+The deployment uses environment-specific configurations:
+- `infrastructure/metabase/environment-configuration.sql`: Database connections and admin users. **This file does not need to be changed**.
+- Environment variables for database credentials and site settings
+- Map configurations for geographic visualizations
+
+## Development and Dashboard Management
+
+### ⚠️ Critical: Development vs Production Changes
+
+**Important**: Changes made directly in deployed Metabase environments (staging, production) **WILL NOT PERSIST** and will be reset during the next deployment.
+
+To make persistent dashboard changes:
+
+1. **Work in Development Mode**: Always make changes in your local development environment first
+2. **Modify the Source**: Update `infrastructure/metabase/metabase.init.db.sql` by running and using Metabase locally. When you stop the local metabase, all Metabase configuration is stored to this file.
+3. **Version Control**: Commit changes to ensure they're deployed to all environments
+4. **Deploy**: Changes will only persist across environments when included in the initialization database
+
+### Development Workflow
+
+```bash
+# Start Metabase in development mode
+yarn metabase
+
+# Access at http://localhost:4444
+# Default credentials:
+# Username: [email protected]
+# Password: m3tabase
+```
+
+### Making Dashboard Changes
+
+1. **Create/modify dashboards** in development Metabase UI
+2. **Export the changes** by updating stopping the process
+3. **Commit changes** to version control
+4. **Deploy** to propagate changes to all environments
+
+### Development Commands
+
+```bash
+# Start Metabase in development
+yarn metabase
+
+# Clear all analytics data
+yarn db:clear:all
+```
+## Why Analytics Data is Not Backed Up
+
+Analytics data in OpenCRVS is **intentionally not included in backup procedures** for several important reasons:
+
+### 1. **Regenerative Nature**
+Analytics data can be completely regenerated from the primary data sources. The analytics tables are derived views of the operational data, not the source of truth.
+
+### 2. **Performance Considerations**
+- Analytics databases can be very large (GB to TB scale)
+- Including them in backups would significantly increase backup time and storage requirements
+- Restore operations would be much slower
+
+## Configuration
+
+### Database Connection
+Metabase connects to PostgreSQL using these environment variables:
+- `METABASE_DATABASE_HOST`: Database host (default: postgres)
+- `METABASE_DATABASE_PORT`: Database port (default: 5432)
+- `METABASE_DATABASE_NAME`: Database name (default: events)
+- `METABASE_DATABASE_USER`: Database user (default: events_analytics)
+- `METABASE_DATABASE_PASSWORD`: Database password
+
+### Analytics Schema
+The analytics data is stored in the PostgreSQL `analytics` schema, primarily in:
+- `analytics.event_actions`: Main table containing processed event data and action histories
+
+### Map Visualizations
+Geographic visualizations use:
+- `OPENCRVS_METABASE_MAP_NAME`: Map display name
+- `OPENCRVS_METABASE_MAP_URL`: GeoJSON source URL
+- `OPENCRVS_METABASE_MAP_REGION_KEY`: Key field for geographic regions
+- `OPENCRVS_METABASE_MAP_REGION_NAME`: Display name for regions
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -15,6 +15,7 @@
 
   - Use nvm to upgrade your local development environment to use node version `22.x.x.`
   - Add conditions for the certified copy certificate to ensure it's only available to children who are 1 year or older. [#9684](https://github.com/opencrvs/opencrvs-core/issues/9684)
+  - Available disk space in root file system alert adjusted to fire when 20GB are remaining, rather than when diskspace usage is at 70%.
 
 - **Upgraded MinIO** to RELEASE.2025-06-13T11-33-47Z and MinIO Client (mc) to RELEASE.2025-05-21T01-59-54Z and ensured compatibility across both amd64 and arm64 architectures.
 

diff --git a/infrastructure/metabase/README.md b/infrastructure/metabase/README.md
@@ -2,13 +2,13 @@
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 **Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
 
-- [OpenCRVS Dashboards](#opencrvs-dashboards)
+- [OpenCRVS Analytics](#opencrvs-analytics)
     - [Run in development mode](#run-in-development-mode)
     - [Default credentials](#default-credentials)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
-# OpenCRVS Dashboards
+# OpenCRVS Analytics
 
 ### Requirements
 

diff --git a/infrastructure/monitoring/kibana/config.ndjson b/infrastructure/monitoring/kibana/config.ndjson
@@ -1,4 +1,4 @@
-{"attributes":{"actions":[{"actionRef":"preconfigured:preconfigured-alert-history-es-index","actionTypeId":".index","group":"metrics.inventory_threshold.fired","params":{"documents":["{\"@timestamp\":\"2024-08-06T07:57:35.644Z\",\"tags\":\"{{rule.tags}}\",\"rule\":{\"id\":\"{{rule.id}}\",\"name\":\"{{rule.name}}\",\"params\":{\"{{rule__type}}\":\"{{params}}\"},\"space\":\"{{rule.spaceId}}\",\"type\":\"{{rule.type}}\"},\"kibana\":{\"alert\":{\"id\":\"{{alert.id}}\",\"context\":{\"{{rule__type}}\":\"{{context}}\"},\"actionGroup\":\"{{alert.actionGroup}}\",\"actionGroupName\":\"{{alert.actionGroupName}}\"}},\"event\":{\"kind\":\"alert\"}}"]}}],"alertTypeId":"metrics.alert.inventory.threshold","apiKey":null,"apiKeyOwner":null,"consumer":"alerts","createdAt":"2023-11-17T12:01:46.420Z","createdBy":"elastic","enabled":false,"executionStatus":{"error":null,"lastExecutionDate":"2024-08-06T08:02:44.568Z","status":"pending"},"legacyId":null,"meta":{"versionApiKeyLastmodified":"7.17.18"},"muteAll":false,"mutedInstanceIds":[],"name":"Available disk space in root file system","notifyWhen":"onActionGroupChange","params":{"alertOnNoData":true,"criteria":[{"comparator":">=","customMetric":{"aggregation":"max","field":"system.filesystem.used.pct","id":"alert-custom-metric","type":"custom"},"metric":"custom","threshold":[0.7],"timeSize":1,"timeUnit":"h","warningComparator":">=","warningThreshold":[0.5]}],"filterQuery":"{\"bool\":{\"should\":[{\"match_phrase\":{\"system.filesystem.mount_point\":\"/hostfs\"}}],\"minimum_should_match\":1}}","filterQueryText":"system.filesystem.mount_point : \"/hostfs\"","nodeType":"host","sourceId":"default"},"schedule":{"interval":"1h"},"scheduledTaskId":null,"tags":["infra","opencrvs-builtin"],"throttle":null,"updatedAt":"2024-08-06T08:01:56.542Z","updatedBy":"elastic"},"coreMigrationVersion":"7.17.18","id":"14778650-8541-11ee-9002-2f37fdc4e5d5","migrationVersion":{"alert":"7.16.0"},"references":[],"sort":[1722931316554,643],"type":"alert","updated_at":"2024-08-06T08:01:56.554Z","version":"WzM5MywxXQ=="}
+{"attributes":{"actions":[{"actionRef":"preconfigured:preconfigured-alert-history-es-index","actionTypeId":".index","frequency":{"notifyWhen":"onActionGroupChange","summary":false,"throttle":null},"group":"metrics.inventory_threshold.fired","params":{"documents":["{\"@timestamp\":\"2025-10-03T09:15:09.124Z\",\"tags\":\"{{rule.tags}}\",\"rule\":{\"id\":\"{{rule.id}}\",\"name\":\"{{rule.name}}\",\"params\":{\"{{rule__type}}\":\"{{rule.params}}\"},\"space\":\"{{rule.spaceId}}\",\"type\":\"{{rule.type}}\"},\"kibana\":{\"alert\":{\"id\":\"{{alert.id}}\",\"context\":{\"{{rule__type}}\":\"{{context}}\"},\"actionGroup\":\"{{alert.actionGroup}}\",\"actionGroupName\":\"{{alert.actionGroupName}}\"}},\"event\":{\"kind\":\"alert\"}}"]},"uuid":"358daf99-65eb-4ff5-a8a5-7218573d60cd"}],"alertTypeId":"metrics.alert.inventory.threshold","apiKey":null,"apiKeyCreatedByUser":null,"apiKeyOwner":null,"consumer":"alerts","createdAt":"2023-11-17T12:01:46.420Z","createdBy":"elastic","enabled":false,"lastRun":{"alertsCount":{"active":0,"ignored":0,"new":0,"recovered":1},"outcome":"succeeded","outcomeMsg":null,"outcomeOrder":0,"warning":null},"legacyId":null,"meta":{"versionApiKeyLastmodified":"8.14.3"},"muteAll":false,"mutedInstanceIds":[],"name":"Available disk space in root file system","notifyWhen":null,"params":{"alertOnNoData":true,"criteria":[{"comparator":"<","customMetric":{"aggregation":"max","field":"system.filesystem.available","id":"alert-custom-metric","type":"custom"},"metric":"custom","threshold":[21474836480],"timeSize":1,"timeUnit":"h"}],"filterQuery":"{\"bool\":{\"should\":[{\"match_phrase\":{\"system.filesystem.mount_point\":\"/hostfs\"}}],\"minimum_should_match\":1}}","filterQueryText":"system.filesystem.mount_point : \"/hostfs\"","nodeType":"host","sourceId":"default"},"revision":5,"running":false,"schedule":{"interval":"1h"},"scheduledTaskId":null,"snoozeSchedule":[],"tags":["infra","opencrvs-builtin"],"throttle":null,"updatedAt":"2025-10-03T09:19:37.847Z","updatedBy":"elastic"},"coreMigrationVersion":"8.8.0","created_at":"2025-10-03T09:19:37.847Z","id":"14778650-8541-11ee-9002-2f37fdc4e5d5","managed":false,"references":[],"type":"alert","typeMigrationVersion":"10.1.0","updated_at":"2025-10-03T09:19:37.847Z","version":"WzExNjc4ODIsMjIyXQ=="}
 {"attributes":{"actions":[{"actionRef":"preconfigured:preconfigured-alert-history-es-index","actionTypeId":".index","group":"threshold_met","params":{"documents":["{\"@timestamp\":\"2022-04-18T07:05:33.819Z\",\"tags\":\"{{rule.tags}}\",\"rule\":{\"id\":\"{{rule.id}}\",\"name\":\"{{rule.name}}\",\"params\":{\"{{rule__type}}\":\"{{params}}\"},\"space\":\"{{rule.spaceId}}\",\"type\":\"{{rule.type}}\"},\"kibana\":{\"alert\":{\"id\":\"{{alert.id}}\",\"context\":{\"{{rule__type}}\":\"{{context}}\"},\"actionGroup\":\"{{alert.actionGroup}}\",\"actionGroupName\":\"{{alert.actionGroupName}}\"}},\"event\":{\"kind\":\"alert\"}}"]}}],"alertTypeId":"apm.error_rate","apiKey":null,"apiKeyOwner":null,"consumer":"alerts","createdAt":"2022-06-01T11:30:27.033Z","createdBy":"opencrvs-admin","enabled":false,"executionStatus":{"error":null,"lastExecutionDate":"2024-02-07T08:28:08.400Z","status":"pending"},"legacyId":null,"meta":{"versionApiKeyLastmodified":"7.17.0"},"muteAll":false,"mutedInstanceIds":[],"name":"Error in service","notifyWhen":"onActionGroupChange","params":{"environment":"ENVIRONMENT_ALL","threshold":1,"windowSize":1,"windowUnit":"m"},"schedule":{"interval":"1m"},"scheduledTaskId":null,"tags":[],"throttle":null,"updatedAt":"2024-02-05T03:00:20.633Z","updatedBy":"opencrvs-admin"},"coreMigrationVersion":"7.17.0","id":"3b6722e0-e19e-11ec-ba8e-51649755648d","migrationVersion":{"alert":"7.16.0"},"references":[],"sort":[1707273006619,214975],"type":"alert","updated_at":"2024-02-07T02:30:06.619Z","version":"WzQ5MjAzNCwxOV0="}
 {"attributes":{"buildNum":46534,"defaultIndex":"metricbeat-*"},"coreMigrationVersion":"7.17.0","id":"7.17.0","migrationVersion":{"config":"7.13.0"},"references":[],"sort":[1707273006619,216009],"type":"config","updated_at":"2024-02-07T02:30:06.619Z","version":"WzQ5MjQyOCwxOV0="}
 {"attributes":{"actions":[{"actionRef":"preconfigured:preconfigured-alert-history-es-index","actionTypeId":".index","frequency":{"notifyWhen":"onActionGroupChange","summary":false,"throttle":null},"group":"logs.threshold.fired","params":{"documents":["{\"@timestamp\":\"2024-12-23T09:50:26.528Z\",\"tags\":\"{{rule.tags}}\",\"rule\":{\"id\":\"{{rule.id}}\",\"name\":\"{{rule.name}}\",\"params\":{\"{{rule__type}}\":\"{{rule.params}}\"},\"space\":\"{{rule.spaceId}}\",\"type\":\"{{rule.type}}\"},\"kibana\":{\"alert\":{\"id\":\"{{alert.id}}\",\"context\":{\"{{rule__type}}\":\"{{context}}\"},\"actionGroup\":\"{{alert.actionGroup}}\",\"actionGroupName\":\"{{alert.actionGroupName}}\"}},\"event\":{\"kind\":\"alert\"}}"]},"uuid":"1504b514-82e5-47de-a6ef-60795788f7e6"}],"alertTypeId":"logs.alert.document.count","apiKey":null,"apiKeyCreatedByUser":null,"apiKeyOwner":null,"consumer":"alerts","createdAt":"2024-12-23T10:16:28.758Z","createdBy":"opencrvs-admin","enabled":false,"executionStatus":{"error":null,"lastExecutionDate":"2024-12-23T10:20:10.118Z","status":"pending","warning":null},"lastRun":{"alertsCount":{"active":1,"ignored":0,"new":1,"recovered":0},"outcome":"succeeded","outcomeMsg":null,"outcomeOrder":0,"warning":null},"legacyId":null,"meta":{"versionApiKeyLastmodified":"8.16.4"},"monitoring":{"run":{"calculated_metrics":{"p50":529,"p95":565,"p99":565,"success_ratio":1},"history":[{"duration":493,"success":true,"timestamp":1734948992101},{"duration":565,"success":true,"timestamp":1734949097318}],"last_run":{"metrics":{"duration":565,"gap_duration_s":null,"total_alerts_created":null,"total_alerts_detected":null,"total_indexing_duration_ms":null,"total_search_duration_ms":null},"timestamp":"2024-12-23T10:18:17.318Z"}}},"muteAll":false,"mutedInstanceIds":[],"name":"Error while backup  ","nextRun":"2024-12-23T11:18:17.247Z","notifyWhen":null,"params":{"count":{"comparator":"more than or equals","value":1},"criteria":[{"comparator":"equals","field":"log.file.path","value":"/var/log/opencrvs-backup.error.log"}],"logView":{"logViewId":"log-view-reference-0","type":"log-view-reference"},"timeSize":1,"timeUnit":"h"},"revision":1,"running":false,"schedule":{"interval":"1h"},"scheduledTaskId":null,"snoozeSchedule":[],"tags":[],"throttle":null,"updatedAt":"2024-12-23T10:17:24.571Z","updatedBy":"opencrvs-admin"},"coreMigrationVersion":"8.8.0","created_at":"2024-12-23T10:17:24.572Z","id":"8caf3676-e78a-4e7c-ba85-c150df55901b","managed":false,"references":[{"id":"default","name":"param:log-view-reference-0","type":"infrastructure-monitoring-log-view"}],"sort":[1734949097887,115],"type":"alert","typeMigrationVersion":"10.1.0","updated_at":"2024-12-23T10:18:17.887Z","version":"WzgyNDExMyw0MDRd"}

diff --git a/infrastructure/monitoring/metricbeat/metricbeat-rollover-policy.json b/infrastructure/monitoring/metricbeat/metricbeat-rollover-policy.json
@@ -1,48 +1,23 @@
 {
   "policy": {
-      "phases": {
-        "hot": {
-          "actions": {
-            "rollover": {
-              "max_size": "2GB",
-              "max_age": "1d"
-            },
-            "set_priority": {
-              "priority": 100
-            }
+    "phases": {
+      "hot": {
+        "min_age": "0ms",
+        "actions": {
+          "rollover": {
+            "max_age": "3d",
+            "max_size": "200mb"
           }
-        },
-        "warm": {
-          "min_age": "3d",
-          "actions": {
-            "allocate": {
-              "number_of_replicas": 0
-            },
-            "forcemerge": {
-              "max_num_segments": 1
-            },
-            "set_priority": {
-              "priority": 50
-            }
-          }
-        },
-        "cold": {
-          "min_age": "7d",
-          "actions": {
-            "allocate": {
-              "number_of_replicas": 0
-            },
-            "set_priority": {
-              "priority": 0
-            }
-          }
-        },
-        "delete": {
-          "min_age": "30d",
-          "actions": {
-            "delete": {}
+        }
+      },
+      "delete": {
+        "min_age": "2d",
+        "actions": {
+          "delete": {
+            "delete_searchable_snapshot": true
           }
         }
+      }
     },
     "_meta": {
       "managed": true,

diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@opencrvs/countryconfig",
-  "version": "1.8.1",
+  "version": "1.9.0",
   "description": "OpenCRVS country configuration for reference data",
   "os": [
     "darwin",
@@ -69,7 +69,7 @@
     "@hapi/boom": "^9.1.1",
     "@hapi/hapi": "^20.0.1",
     "@hapi/inert": "^6.0.3",
-    "@opencrvs/toolkit": "1.8.1-rc.c8c6227",
+    "@opencrvs/toolkit": "1.9.0",
     "@types/chalk": "^2.2.0",
     "@types/csv2json": "^1.4.0",
     "@types/fhir": "^0.0.30",