feat: Provide way to configure alerting rules

[adskyiproger]

Testing

Made changes on existing helm chart

Modify rule by file size instead of percentage:

Navigate to "Stack management" https://kibana.tmp-stg.opencrvs.dev/app/management

Navigate to "Saved Objects": Export all modified objects. This file will be used on the second part of the testing process to test custom kibana configuration.

Redeploy dependencies: https://github.com/opencrvs/infrastructure/actions/runs/20159680591

Make sure settings was discarded:

Testing custom kibana configuration

Dump helm release values:

helm get values opencrvs-deps > opencrvs-deps.yaml

values.yaml example:

#### this is custom configmap with exported alerting rules
kibana:
  custom_config_configmap: kibana-custom-config
#### this is custom configmap with exported alerting rules
elastalert:
  env:
    HTTP_POST2_ALERT_URL: http://countryconfig.opencrvs-staging.svc.cluster.local:3040/email
elasticsearch:
  use_default_credentials: false
environment_type: production
hostname: tmp-stg.opencrvs.dev
minio:
  use_default_credentials: false
mongodb:
  use_default_credentials: false
monitoring:
  enabled: true
postgres:
  use_default_credentials: false
redis:
  auth_mode: acl
restore:
  backup_encryption_secret: restore-encryption-secret
  backup_server_dir: /home/backup/production
  backup_server_secret: backup-server-ssh-credentials
  enabled: true
  schedule: 0 0 * * *
storage_type: host_path

Create custom configmap from config.ndjson exported earlier:

kubectl create cm kibana-custom-config --from-file config.ndjson

Deploy helm chart from custom branch:

helm upgrade -f ./opencrvs-deps.yaml opencrvs-deps ./charts/dependencies/

Make sure modified alerting rule was displayed in Kibana: