Security and Maintenance

Please refer to Terra UI Maintenance regarding the maintenance rotation and the related responsibilities.

Service Account Key Rotation

Every three months, the service account key used by CircleCI to deploy Terra UI to prod needs to be rotated for security purposes, by the maintaining team. Follow the steps below to create a new key and deploy it.

NOTE: This is different from the TERRA_SA_KEY used in the integration tests, if this needs to be rotated, follow the same steps as below to create a new key, but for the firecloud-dev SA and the general Terra-UI env variables. Full list of CircleCi vars can be found here

Create new key in Google Cloud Console

• Having logged in with your @firecloud.org account, go to Google Cloud Console's CircleCI Service Account Keys page of the IAM & Admin section for project BVDP Saturn Prod. If you receive a permissions error, ask on #terra-ui Slack channel to be granted Editor role on Google project BVDP Saturn Prod.
• Click on the ADD KEY button.
• Select Create new key.
• Make sure JSON is selected as key type, and click on CREATE.
• Make sure the .json file was downloaded onto your computer.

Update Circle CI's GCP_SA_KEY environment Variable

• Create a new browser tab, and go to CircleCI page for saturn-ui-prod-deploy project. In here you can find Environment Variables under Settings. If you can't view this page, ask on #terra-ui Slack channel to be added to GitHub team SaturnDeployers.
• Delete the environment variable named GCP_SA_KEY by clicking on X to the right of it.
• Click on the button Add Environment Variable and fill out Name and Value on the dialog that pops up.
  • Name should be GCP_SA_KEY
  • Value is supposed to be the text content of the .json file you had downloaded.
• Manually redeploy to prod to verify the new SA key works by following the instructionshere.
• Observe on the saturn-ui-prod-deploy CircleCI page that the job has succeeded.
• Go to https://app.terra.bio/#workspaces; click on the hamburger menu on the upper left; click on the link to the right of Built on: on the bottom; verify that the latest GitHub commit is on the new tab that pops up.

Disable and delete the old deployment key

The recommended flow here is:

• Disable the old key
  • See note below
• Test that circle CI still works
• Delete the old key
  • Use the trash can icon in the GCP console

Disabling the old key

It's not possible to disable a SA key on the GCP console. You will need to use gcloud on your local system (e.g. Terminal on MacOS). After making sure you're authenticated with your @firecloud.org account on your local system, run gcloud iam service-accounts keys disable <service_account_key_name> [email protected] --project bvdp-saturn-prod. Substitute into that command, the key obtained from Key column on the GCP console page for SA keys. More details on disabling service account keys can be found here. If you get ERROR: (gcloud.iam.service-accounts.keys) Invalid choice: 'disable'., try upgrading your SDK via gcloud components update. Verify on the GCP console page that the key's Status is now Disabled (as opposed to Active).

Prepare for the next rotation

• Set a Slack reminder on #terra-ui channel for the next maintaining team by typing something like /remind :alert_red_light: Reminder to rotate the service account key for prod deploys by following these (https://github.com/DataBiosphere/terra-ui/wiki/Security-and-Maintenance#service-account-key-rotation) instructions in 85 days.

Update Vault with the new key

Note: This step is not required and may even be an anti-pattern. If a new CI key is required, it is better to follow these steps to generate a new one rather than try to recover one out of some cold-storage.

Add the new .json key file (that you downloaded at the beginning) to Vault (e.g. vault write secret/dsp/service-accts/bvdp-saturn-prod-sa-2021-12-14.json @/Users/<your username>/Downloads/<new sa file name>.json).

Dependabot Security Updates

Anyone with write access to Terra UI should be able to view Dependabot alerts. To receive notifications for new Dependabot alerts, you must have notifications for alerts turned on and must be watching the Terra UI repository (either all activity or custom events including security alerts). See Access to Dependabot alerts for more information.

A lot of the Dependabot alerts can be safely dismissed. These are all valid reasons for dismissing the alerts:

• The alert is for a library version that is only used by integration tests, development tools or a library that is only used in the packaging of our code (for example, packaging CSS), and thus not exposed to users. To figure this out, use yarn why -R LIBRARY_NAME. For instance, if you see a root node name in the yarn why output that reads terra-integration-tests@workspace:integration-tests, we know that the library is used by integration-tests. yarn why -R --json LIBRARY_NAME | jq '.value' can also be helpful in seeing the root packages at a glance without the entire tree.
• The alert is incorrect.
  • The alert is based on integration-tests/yarn.lock (as opposed to yarn.lock at the root of the repo) and the package is not listed as a direct dependency on integration-tests/package.json. Dependabot has a bug where it may continue alerting based on files that no longer exists in a repo. This has happened for this repo with integration-tests/yarn.lock that had been removed.
  • By examining the yarn.lock file and using yarn why LIBRARY_NAME it is clear that we actually don't use the library version that the alert is warning about.

If the library version does need to be upgraded because it is exposed to users in Terra's UI, these are the recommended steps:

• If the library is one of our direct dependencies, update the version in package.json and generate an updated yarn.lock file as detailed below.
• If the library is a transitive dependency, delete the entries for the library in yarn.lock.
• Make sure running yarn --version outputs the same version as the one here.
• Run yarn install to generate the new content in yarn.lock. Verify the library versions satisfy the security update.
• Run yarn dedupe to remove duplicated content from the yarn.lock file.
• Create a PR with the modified .pnp.js, *.zip, and yarn.lock files.
• Extra points for updating our direct dependencies (package.json files) that introduce the transitive dependency to their most recent versions 🥇 😃

Compliance timeframe for addressing security alerts can be found in Terra Vulnerability Management Plan.

Adjusting Prod Deploy Time for Daylight Savings Time Changes

Prod deploy time has been chosen as 11am Eastern Time. See this PR for start of DST and this PR for end of DST.

Adjusting Production Deploy Schedule for Holidays

To reduce the chance of an incident over a holiday, we avoid releasing Terra UI on Broad holidays. The list of holidays in the production deploy workflow configuration needs to be updated each year since some holidays will fall on different dates.

For reference, see PRs from 2022 and 2023.

Cleaning up old non-prod App Engine versions

Terra UI is a Google App Engine application. Every time CircleCI deploys Terra UI, it creates a new version. App Engine allows a maximum of 210 versions of any app. Therefore, we must periodically clean up old versions.

In prod there is an automated script to handle the cleanup in CircleCI. In non-prod (dev, alpha, staging) there is a GitHub Action that runs every week. If there's an issue, you can also manually run the Github Action with the Run workflow button.

In the worst case, if you need to debug the script locally:

1. Ensure you have jq installed beforehand.
2. Ensure that you are logged in to the correct gcloud account that has App Engine deletion permission.
3. git clone the Terra UI repository or download the script from https://github.com/DataBiosphere/terra-ui/tree/dev/scripts.
4. Run the delete-old-app-engine-versions.sh script and specify the environment to delete, which will prompt you to confirm deletion of old app engine resources in that environment.

You're done! Thank you for your service!

Adding PR Sites