updated readme

Author: Vadym Mudryi

README.md

@@ -112,7 +112,7 @@ You need to clone the [opencrvs-core](https://github.com/opencrvs/opencrvs-core)
     ```
 6. [Temporary Step] Switch to the k8s-version branch:
     ```bash
-    git checkout k8s-version
+    git checkout k8s-refresh
     ```
 7. Run Tilt:
     ```bash
@@ -123,27 +123,55 @@ You need to clone the [opencrvs-core](https://github.com/opencrvs/opencrvs-core)
 
 ---
 
-## [🚧  Coming soon] For OpenCRVS Country Configuration Developers
+## For OpenCRVS Country Config Developers
 
-You need to fork the [opencrvs-countryconfig](https://github.com/opencrvs/opencrvs-countryconfig) repository and clone the [infrastructure](https://github.com/opencrvs/infrastructure) repository. If these repositories are already on your laptop, ensure they are in the same folder.
+Please follow official documentation how to setup your own country configuration at [Set-up your own, local, country configuration](https://documentation.opencrvs.org/setup/3.-installation/3.2-set-up-your-own-country-configuration).
+You need to fork (clone) the [opencrvs-countryconfig](https://github.com/opencrvs/opencrvs-countryconfig) repository and clone the [infrastructure](https://github.com/opencrvs/infrastructure) repository. If repositories are already on your laptop, ensure they are in the same parent folder, for example:
+```
+repositories/
+    infrastructure
+    opencrvs-countryconfig
+    ...
+```
 
-1. Create a new folder or use an existing folder to store the repositories.
+**Step by step instruction**
+
+1. Create a new folder or use an existing folder to store the repositories. For example folder could be located at your home directory or in documents:
+   ```bash
+   mkdir ~/Documents/repository
+   ```
 2. Open a terminal (command line) and navigate to the folder.
-3. Clone your fork of the OpenCRVS Country Configuration repository:
+   ```bash
+   cd ~/Documents/repository
+   ```
+3. Clone OpenCRVS Country Config repository:
+    
+    For county config use:
+    ```bash
+    git clone https://github.com/opencrvs/opencrvs-countryconfig
+    ```
+    For your own fork use:
     ```bash
     git clone [email protected]:<your-github-account>/<your-repository>.git
     ```
+
 4. Clone the Infrastructure repository:
     ```bash
     git clone [email protected]:opencrvs/infrastructure.git
     ```
-5. Change directory to your forked repository:
+5. Change directory to country config (your own) repository:
+    
+    For county config use:
+    ```bash
+    cd opencrvs-countryconfig
+    ```
+    For your own fork use:
     ```bash
     cd <your-repository>
     ```
-6. [Temporary Step] Switch to the k8s-version branch:
+6. [Temporary Step] Switch to the k8s-refresh branch:
     ```bash
-    git checkout k8s-version
+    git checkout k8s-refresh
     ```
 7. Run Tilt:
     ```bash
@@ -152,6 +180,16 @@ You need to fork the [opencrvs-countryconfig](https://github.com/opencrvs/opencr
 8. Navigate to [http://localhost:10350/](http://localhost:10350/)
 9. Once all container images are up and running your environment will be available at https://opencrvs.localhost
 
+## Seed data
+
+1. Navigate to file `kubernetes/opencrvs-services/values-dev.yaml` in opencrvs-core (or your country config) repository
+2. Change value `data_seeder.enabled` to `true`.
+3. Save changes
+4. New tilt resource `data-seeder` will be created, check [http://localhost:10350/](http://localhost:10350/)
+5. Make sure data-seeder job completed without issues.
+6. Change value `data_seeder.enabled` to `false`.
+7. Save changes
+
 ## Common issues
 
 ### Container start is failing with ImagePullBackOff
@@ -162,7 +200,53 @@ Check image tag was set properly, use `kubectl`, adjust value in `kubernetes/ope
 
 ### Reset local environment
 
-Restart docker desktop
+Draft and working way is to restart docker desktop
+
+### Troubleshooting connectivity inside Kubernetes cluster
+
+1. Issue fresh token:
+
+  ```bash
+  USERNAME=o.admin
+  SUPER_USER_PASSWORD=password
+  curl -X POST "http://auth.opencrvs-dev.svc.cluster.local:4040/authenticate-super-user" \
+      -H "Content-Type: application/json" \
+      -d '{
+        "username": "'"${USERNAME}"'",
+        "password": "'"$SUPER_USER_PASSWORD"'"
+      }'
+  ```
+
+2. Check gateway host:
+  ```bash
+    GATEWAY_HOST=http://gateway.opencrvs-dev.svc.cluster.local:7070
+    curl -X GET \
+        -H "Content-Type: application/json" \
+        -H "Authorization: Bearer ${token}" \
+        ${GATEWAY_HOST}/locations?type=ADMIN_STRUCTURE&_count=0
+  ```
+3. Check config host:
+  ```bash
+  curl -v -X GET \
+      -H "Content-Type: application/json" \
+      -H "Authorization: Bearer ${token}" \
+      http://config.opencrvs-dev.svc.cluster.local:2021/locations?type=ADMIN_STRUCTURE&_count=0
+  ```
+4. Check Hearth:
+  ```bash
+  curl -v http://hearth.opencrvs-deps-dev.svc.cluster.local:3447/fhir/Location
+  ```
+
+### Login/Client service is not responding: Check login logs
+```
+2025/03/19 07:53:38 [error] 15#15: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.1.3.102, server: localhost, request: "GET /api/countryconfig/login-config.js HTTP/1.1", upstream: "http://10.100.14.175:3040/login-config.js", host: "login.opencrvs.localhost", referrer: "https://login.opencrvs.localhost/"
+```
+
+Solution: restart nginx inside login container or delete login pod
+```
+nginx -s reload
+```
+
 
 ---
 

charts/opencrvs-services/README.md

@@ -18,7 +18,7 @@ Helm chart to deploy all OpenCRVS services on Kubernetes cluster.
         <tr>
             <td>elasticsearch_host</td>
             <td>elasticsearch.opencrvs-deps-dev.svc.cluster.local:9200</td>
-            <td>Elasticsearch configuration, including the hostname and port. TODO: Consider defining the port as a separate variable.</td>
+            <td>Elasticsearch configuration, including the hostname and port.<br> <b>NOTE</b>: Some services require authentication, please use secrets to redefine ES_HOST variable if needed.</td>
         </tr>
         <tr>
             <td>influxdb.host</td>
@@ -55,6 +55,11 @@ Helm chart to deploy all OpenCRVS services on Kubernetes cluster.
             <td>mongodb-0.mongodb.opencrvs-deps-dev.svc.cluster.local</td>
             <td>MongoDB hostname configuration.</td>
         </tr>
+        <tr>
+            <td>redis_host</td>
+            <td>redis-0.redis.opencrvs-deps-dev.svc.cluster.local</td>
+            <td>Redis hostname configuration.</td>
+        </tr>
         <tr>
             <td>hostname</td>
             <td>farajaland.com</td>
@@ -80,26 +85,17 @@ Helm chart to deploy all OpenCRVS services on Kubernetes cluster.
             <td>{}</td>
             <td>Mapping kubernetes secrets as environment variables. For more information see [Mapping secrets](#mapping-secrets)</td>
         </tr>
+        <tr>
+            <td>data_seeder.enabled</td>
+            <td>true</td>
+            <td>Seed data as post-install step, data seeder is executed only once while `helm install`. In some cases when data is already seeded, e/g upgrade, this value must be set to false. **Note**: default user is used for data seeding, it will fail anyway on database with non-default data.</td>
+        </tr>
     </tbody>
 </table>
 
-# Microservice environment variables configuration
-
-<pre>Do we need this section?</pre>
-
-Helm chart allows to define environment variables in following scopes:
-- **Global variables** are defined at top level of values file and is added to all containers. See `env` key in [values.yaml](values.yaml)
-- **Service level variables** are defined for each particular service. See `<service_name>.env` key in [values.yaml](values.yaml)
-- **Secret environment variables** are defined at service level as `<service_name>.secrets` key, see [values.yaml](values.yaml).
 
 # Mapping secrets
 
-Suppose we need to store ES_HOST variable as a secret since it contains url with login and password for Elastic search.
-Kubernetes secret is key/value object usually created from `.env` file, for example:
-```
-ES_HOST=user:randompass@elasticsearch:9200
-```
-
 Mapping needs to be added for particular service to access variable inside workload (service), e/g for `search` service to access ES_HOST following configuration is needed:
 ```
 search:
@@ -117,4 +113,37 @@ secrets:
 Summary:
 - `secret_name`, name of Kubernetes secret object
 - `secret_key`, key (variable name) inside Kubernetes secret data property
-- `environment_variable`, environment variable name inside container. If `secret_key` value `environment_variable` are the same, last one can be omitted.
+- `environment_variable`, environment variable name inside container. If `secret_key` value `environment_variable` are the same, last one can be omitted.
+
+**Step by step example**
+
+Suppose we need to store ES_HOST variable as a secret and provide variable value to service `search`.
+
+1. Create `.env` like file and put all variables:
+    ```
+    ES_HOST=user:randompass@elasticsearch:9200
+    ```
+2. Create kubernetes secret from `.env` file:
+    ```
+    kubectl create secret generic elasticsearch-secret --from-env-file=.env
+    ```
+3. Make sure the secret was created:
+    ```
+    kubectl get secret -oyaml elasticsearch-secret
+    ```
+    Example output:
+    ```yaml
+    apiVersion: v1
+    data:
+        ES_HOST: dXNlcjpyYW5kb21wYXNzQGVsYXN0aWNzZWFyY2g6OTIwMA==
+    ...
+    ```
+3. Map variable in your helm chart values file:
+    ```yaml
+    search:
+        secrets:
+            elasticsearch-secret:
+                - ES_HOST
+    ...
+    ```
+4. Redeploy service with `helm upgrade`

charts/opencrvs-services/Troubleshooting inside Kubernetes cluste.md

@@ -25,7 +25,5 @@ spec:
             - name: COUNTRY_CONFIG_HOST
               value: http://countryconfig.{{ .Release.Namespace }}.svc.cluster.local:3040
           {{- include "render-env-vars" (dict "service_name" "data_seeder" "Values" .Values) }}
-      completions: 1 # Run the job once
-      backoffLimit: 0 # Don't retry if the job fails
       restartPolicy: "OnFailure"
 {{- end }}

charts/opencrvs-services/templates/data-seeder.yaml

@@ -44,7 +44,4 @@ spec:
             - name: WAIT_HOSTS
               value: "{{ .Values.mongodb_host }}:27017,{{ .Values.influxdb.host }}:{{ .Values.influxdb.port }},{{ .Values.minio.host }}:{{ .Values.minio.port }},{{ .Values.elasticsearch_host }}"
           {{- include "render-env-vars" (dict "service_name" "migration" "Values" .Values) }}
-      # TODO: Check if these properties are required for the migration job.
-      completions: 1 # Run the job once
-      backoffLimit: 0 # Don't retry if the job fails
       restartPolicy: "OnFailure"

charts/opencrvs-services/templates/migration-job.yaml

@@ -56,8 +56,7 @@ config:
     CHECK_INVALID_TOKEN: true
     NODE_ENV: production
   # Mapping for external secrets goes here:
-  
-  secrets:
+  secrets: {}
 
 countryconfig:
   image: