As of the January 2024 deployment of the BIP shared microservice, VRO is only using the BIP Claims API. While it no longer uses the BIP Claim Evidence API, the codebase for the Claims Evidence API is still present. - [BIP Claims API](BIP%20Claims%20API) The above pages contain information specific to one of the individual APIs. This page provides these links and describes the details of [VRO Mutual TLS](#mutual-tls-mtls) implementation. We will dive into the details of the TLS certificates momentarily, but it is first helpful to understand the big picture of how these certs are used. First, there is the [certificate management system](#certificate-management) that acts as the VA's primary Certificate Authority (CA). Our CA is provided by LHDI, a Venafi system deployed as a feature of our Kubernetes infrastructure. Through Venafi, we provision and query for certificates, and once provisioned, Venafi manages the expiration of these certificates. Second, there is the generation process itself which is a bash script that lives in the VRO repo: [scripts/svc-bip-api-certgen.sh](https://github.com/department-of-veterans-affairs/abd-vro/blob/142ba71fe6afe2afd5ca9326ac3c0b3bfca73aad/scripts/svc-bip-api-certgen.sh). This script pulls the certs from Venafi and combines them with the intermediate certificates which it also downloads. The certs from Venafi, along with the Intermediate certificates, are then crunched through a series of OpenSSL commands to produce an output.json file ultimately. Third, there is [HashiCorp Vault](./Secrets-Vault.md), where we store the generated truststore, keystore, and the password used to seal these stores. Fourth and finally, the Kubernetes secrets supply the application with the trust store, key store, and password. Please reference the following diagram as you navigate through the remainder of this documentation: ![](images/mututal-tls-certs.png) (note: this image was created with draw.io - the png file contains the draw.io source as embedded data to the png file) ## VRO_SECRETS_BIP The secrets that are pertinent to these integrations can be found in [HashiCorp Vault](./Secrets-Vault.md) under `vro-secrets/deploy/{ENV_NAME}/VRO_SECRETS_BIP` and include: * BIP_CLAIM_SECRET * BIP_CLAIM_USERID * BIP_EVIDENCE_SECRET * BIP_EVIDENCE_USERID * BIP_KEYSTORE * BIP_PASSWORD * BIP_TRUSTSTORE Additionally, RabbitMQ and Datadog credentials are also found in Vault under `vro-secrets/deploy/{ENV_NAME}/VRO_SECRETS_MQ` and `vro-secrets/deploy/{ENV_NAME}/VRO_SECRETS_DATADOG` respectively. Secrets provided are: - MQ_CLIENTUSER_NAME - MQ_CLIENTUSER_PASS - DD_SITE - DD_API_KEY - DD_APP_KEY ## Mutual TLS (mTLS) BIP APIs and other VA services require [Mutual TLS, or mTLS](https://en.wikipedia.org/wiki/Mutual_authentication). VRO must use a client certificate signed by a VA-recognized [Certificate Authority](https://en.wikipedia.org/wiki/Certificate_authority) (CA) during [https handshake](https://en.wikipedia.org/wiki/Transport_Layer_Security). In addition, VRO needs to use a VA-recognized [CA]((https://en.wikipedia.org/wiki/Certificate_authority)) to validate BIP certificates during [https handshake](https://en.wikipedia.org/wiki/Transport_Layer_Security). ## Certificate Management [Venafi](https://community.max.gov/display/VAExternal/Requesting+Venafi+Access+and+Creating+Internal+VA+Certificates) is the VA's new Internal Management System, which can be used to manage the PKI certificates. Within the VRO Kubernetes cluster, the process is further simplified, and it is possible to generate the certificates within one of the VRO Kubernetes namespaces. The client certificates that VRO currently uses have been generated using these [instructions](https://animated-carnival-57b3e7f5.pages.github.io/container-platform/cert-management/). The certificates `tls.key`, `tls.crt`, and `va.crt` have been generated in the `va-abd-rrd-dev` namespace and can be found in the `va-abd-rrd-dev-va-gov-tls` secrets. The private key `tls.key` is not encrypted, so no passwords are needed for the steps described later in this document. Details for `tls.crt` can be displayed using ```bash openssl x509 -in tls.crt -text -noout ``` The certificate `tls.crt` expires on January 6th, 2024. The certificate `tls.crt` contains both the client public key and an intermediate CA. Only the client public key is used and referred to as `tls_bip.crt`. The certificate `va.crt` is not currently used. By default, certificates generated are valid for one year and are set to auto-renew ⅔rds of the way through their lifecycle. ### Production Use A second set of certificates is generated in production. This document's discussion of production certificates is identical. The production certificate expires on December 14, 2025. ### Future Work As discussed later in this document, we do not yet know a way to use these certificates directly in the Java code, and there are [manual steps](#manual-steps) to make them usable. More discussions are needed with LHDI around the process of updating them. - The certificates will be auto-renewed. What is the trigger for performing manual steps when the switch is made? - Should LHDI make these certificates available to containers without additional steps? - Can we forego container-level https and move that complexity to API Gateway so that LHDI manages it? At this time, Java cert renewal is manual, and the [steps](#manual-steps) in this document should be followed. ## Downloaded Certificates During the VRO implementation of mTLS, we were not successful using the VA CA public keys in `va.crt` and `tls.crt`. Instead, we downloaded the public keys that were used by BIP APIs from Chrome on a GFE: - Open Chrome on your GFE - Open one of the endpoints in [BIP Claims API](BIP%20Claims%20API). If you use a `GET` endpoint, you should typically get a `jwt` missing error - Once you get the error, click on the `lock` (View site information) in the URL bar to bring up a pop-up window Select the "Connection is secure" menu item to bring up a second pop-up window. From this pop-up, select "Certificate is valid" to bring up the "Certificate viewer." - Go to the "Details" tab, select the certificate for root CA, and "Export". Repeat for the intermediate CA Two certificates downloaded are - VA-Internal-S2-RCA1-v1.crt (root) - VA-Internal-S2-ICA4.crt (intermediate) These public keys are concatenated in a file named `va_all.crt` for later use. ### Future Work We have not retried using the VA CA public keys in `va.crt` and `tls.crt` after we successfully used `va_all.crt`. This should be revisited in the future since it is possible that the steps we eventually ended up with could be successful for public keys in `va.crt` and `tls.crt` as well. ## Self-Signed Certificates VRO generates a set of self-signed certificates using a script called [build-certificates.sh](https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/scripts/build-certificates.sh). These certificates are used for local development and end-to-end tests to mock the actual certificates and [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) based [https handshake](https://en.wikipedia.org/wiki/Transport_Layer_Security). More details and what is being generated are documented in the [script](https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/scripts/build-certificates.sh). # VRO mTLS Implementation ## PKCS #12 Files We do not know a simple way to consume the certificates `tls.key`, `tls_bip.crt`, and `va_all.crt` directly in the Java code to utilize them in `https` calls. Java uses either [JKS](https://en.wikipedia.org/wiki/Java_KeyStore) files or [PKCS #12](https://en.wikipedia.org/wiki/PKCS_12) files. Since [PKCS #12](https://en.wikipedia.org/wiki/PKCS_12) files are not Java specific, we prefer them over [JKS](https://en.wikipedia.org/wiki/Java_KeyStore) files in VRO. To generate the PKCS #12 file `keystore.p12` for client certificates we use [`OpenSSL](https://www.openssl.org/) ```bash openssl pkcs12 -export -in tls_bip.crt -out keystore.p12 -name keystore -nodes -inkey tls.key ``` This command asks for an "export password." The password is to be recorded as it will later be used as a secret. Note that currently the `BIP_KEYSTORE` and `BIP_TRUSTSTORE` values share the same `BIP_PASSWORD` value for those certificates. To generate the PKCS #12 file `truststore.p12` for VA CA certificates we use [`keytool`](https://docs.oracle.com/en/java/javase/17/docs/specs/man/keytool.html) ```bash keytool -import -file va_all.crt -alias all_cas -keystore truststore.p12 ``` This command asks for a "keystore password." For simplicity, VRO uses the same value as the "export password." ### Future Work It should be possible to use `keystore` or `openssl` exclusively here. But these PKCS #12 files worked, and we did not try either route further. ## Kubernetes Secrets VRO stores the content of the [PKCS #12] files and the "export password" in three environment variables. - `BIP_KEYSTORE` - `BIP_TRUSTSTORE` - `BIP_PASSWORD` Since PKCS #12 files are binary, we convert them to [Base 64](https://en.wikipedia.org/wiki/Base64) using `openssl` ```bash openssl base64 -in keystore.p12 -out keystore.b64 openssl base64 -in truststore.p12 -out truststore.b64 ``` What is stored in the Vault secrets `BIP_KEYSTORE` and `BIP_TRUSTSTORE` are the content of the Base 64 files. In our pods, the secrets are injected as environment variables to be available to the `app` container via `application.yaml`: - keystore: ${BIP_KEYSTORE} - truststore: ${BIP_TRUSTSTORE} - truststore_password: ${BIP_PASSWORD} ### Manual Steps In summary, the manual steps to make the certificates available to the Java code are - Convert certificates to PKCS #12 files - Convert PKCS #12 files to Base 64 files - Store content of the Base 64 files in Vault secrets - Hard refresh and sync the `vro-secrets-bip-vault` secrets template in ArgoCD - Restart the pod to make secret changes effective If the `https` handshake implementation remains in the Java code (as opposed to being moved to API Gateway), these manual steps need to be automated if certificate renewal is automated. ## Java implementation and Code Walkthrough Java uses [Keystore](https://docs.oracle.com/javase/8/docs/api/java/security/KeyStore.html) objects to store certificate information in PKCS #12 files. VRO - Reads in the content of the PKCS #12 files and the password from the environment variables `BIP_KEYSTORE`, `BIP_TRUSTSTORE`, and `BIP_PASSWORD` through [application.yml](https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/app/src/main/resources/application.yml) - Converts Base 64 content to binary content - Creates the keystore and trusstore objects as [Keystore](https://docs.oracle.com/javase/8/docs/api/java/security/KeyStore.html) instances from the binary content and the password - Creates a custom [RestTemplate](https://www.baeldung.com/rest-template) [bean](https://www.baeldung.com/spring-bean) that can be used to make the `https` requests The properties in [application.yml](https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/app/src/main/resources/application.yml) that corresponds to the environment variables are - `keystore` for `BIP_KEYSTORE` - `truststore` for `BIP_TRUSTSTORE` - `truststore_password` for `BIP_PASSWORD` The [RestTemplate](https://www.baeldung.com/rest-template) [bean](https://www.baeldung.com/spring-bean) that is used to make the `https` requests is implemented in [BipApiConfig](https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/app/src/main/java/gov/va/vro/config/BipApiConfig.java). This file also includes the generation of the [Keystore](https://docs.oracle.com/javase/8/docs/api/java/security/KeyStore.html) objects from the [application.yml](https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/app/src/main/resources/application.yml) properties. This [RestTemplate](https://www.baeldung.com/rest-template) bean is autowired in the rest of the code using the [Qualifier](https://www.baeldung.com/spring-qualifier-annotation) `bipCERestTemplate`. # Curl Validation of Certificates Validation of the certificates using the Java code has been problematic since the BIP APIs are not available outside of VA Firewall - Our GFE's typically do not have Java installed - Extra permissions are necessary to install and maintain Java on the GFE's - Additional security features in GFE's make it challenging to make https calls from Java [`curl`](https://curl.se/) has been an invaluable tool to test the validity of the certificates. In principle, it should be possible to run `curl` in your GFE, but running it from the `app` container in one of our environments was easier. We copied the certificates `tls.key`, `tls_bip.crt`, and `va_all.crt` to the container using `cat < --cacert va_all.crt --cert tls_bip.crt --key tls.key --verbose ``` To verify the validity of the certificates, see the `https` handshake. The request can be made to any endpoint of interest.