Partner Onboarder: Jobs fail due to unknown MOSIP root CA (v3)

[mbert]

When installing the partner onboarder, at the end the demo jobs are run. In the generated reports it turns out that checks fail. It seems like the jobs try to upload certificates as intermediate CA. However their root CA is not present in the partner management, so that the intermediate certificates are rejected.

It looks to me like this is a general problem not related to the system running in an internal network.

I have eventually been able to work around this problem by downloading the root certificate and then uploading it to the partner management. However the solution is rather ugly, because the partner management services need to be up in order for this to succeed, and as the jobs are run within the same helm call as the installation, there's a bit of a chicken-egg problem.

The solution is to run helm twice: once for installation only, no jobs, then again for the jobs. Hence there's 2 values.yaml files.

values-bootstrap.yaml is for installation only, all jobs disabled (the first 3 blocks are workarounds for the use of a corporate CA as described in #1714, may not be relevant for everybody):

extraEnvVars:
  - name: NODE_EXTRA_CA_CERTS
    value: /etc/ssl/certs/corporate-ca.pem

extraVolumes:
  - name: corporate-ca
    secret:
      secretName: corporate-ca

extraVolumeMounts:
  - name: corporate-ca
    mountPath: /etc/ssl/certs/corporate-ca.pem
    subPath: corporate-ca.pem
    readOnly: true

onboarding:
  modules:
    - name: ida
      enabled: false
    - name: print
      enabled: false
    - name: abis
      enabled: false
    - name: resident
      enabled: false
    - name: digitalcard
      enabled: false

The file values.yaml is identical, just every onboarding module's enabled set to true.

Then split the helm calls in install.sh and add code for downloading / uploading the certificate (the --post-renderer arguments are workarounds for the use of a corporate CA as described in #1714, may not be relevant for everybody):

  echo Install onboarding module
  helm -n $NS install partner-onboarder mosip/partner-onboarder \
  $NFS_OPTION \
  $S3_OPTION \
  --set onboarding.variables.push_reports_to_s3=$push_reports_to_s3 \
  --set extraEnvVarsCM[0]=global \
  --set extraEnvVarsCM[1]=keycloak-env-vars \
  --set extraEnvVarsCM[2]=keycloak-host \
  $ENABLE_INSECURE \
  -f values-bootstrap.yaml \
  --version $CHART_VERSION \
  --wait-for-jobs \
  --post-renderer ./post-render.sh


  API_HOST="$(kubectl -n onboarder get configmap global -o jsonpath='{.data.mosip-api-internal-host}')"
  CLIENT_SECRET="$(kubectl -n onboarder get secret keycloak-client-secrets -o jsonpath='{.data}' | jq | grep mosip_deployment_client_secret | awk '{ print $2 }' | tr -d '",' | base64 -d)"
  AUTH_JSON="{
  \"id\":\"string\",
  \"metadata\":{},
  \"request\":{
    \"appId\":\"ida\",
    \"clientId\":\"mosip-deployment-client\",
    \"secretKey\":\"$CLIENT_SECRET\"
  },
  \"requesttime\":\"2026-04-08T09:43:52.913Z\",
  \"version\":\"string\"
}"
  echo 1>&2
  echo

  echo Download session cookie to headers.txt:
  curl -sk -D headers.txt -H 'Content-Type: application/json' -d "$AUTH_JSON" "https://${API_HOST}/v1/authmanager/authenticate/clientidsecretkey"
  SESSION_COOKIE="$(grep -i '^set-cookie: Authorization=' headers.txt | sed 's/^set-cookie: //I' | cut -d';' -f1)"
  echo 1>&2
  echo

  echo Download root-ca.pem:
  curl -sk -H "Cookie: $SESSION_COOKIE" "https://${API_HOST}/v1/keymanager/getCertificate?applicationId=ROOT" | jq .response.certificate | perl -pe 's/"//g; s/\\n/\n/g' >root-ca.pem
  echo 1>&2
  echo

  echo Upload root-ca.pem, ignore KER-PCM-003 if it is already there:
  curl -sk -X POST "https://${API_HOST}/v1/partnermanager/partners/certificate/ca/upload" -H "Cookie: $SESSION_COOKIE" -H "Content-Type: application/json" -d @- <<EOF
{
  "id": "string",
  "metadata": {},
  "request": {
    "certificateData": "$(awk '{printf "%s\\n", $0}' root-ca.pem)",
    "partnerDomain": "AUTH"
  },
  "requesttime": "$(date -u +%Y-%m-%dT%H:%M:%S.000Z)",
  "version": "string"
}
EOF
  echo 1>&2
  echo

  echo Run onboarding jobs
  helm -n $NS upgrade partner-onboarder mosip/partner-onboarder \
  $NFS_OPTION \
  $S3_OPTION \
  --set onboarding.variables.push_reports_to_s3=$push_reports_to_s3 \
  --set extraEnvVarsCM[0]=global \
  --set extraEnvVarsCM[1]=keycloak-env-vars \
  --set extraEnvVarsCM[2]=keycloak-host \
  $ENABLE_INSECURE \
  -f values.yaml \
  --version $CHART_VERSION \
  --wait-for-jobs \
  --post-renderer ./post-render.sh