From 5e5a232e410b6f08cbd423f753d080e6684048d9 Mon Sep 17 00:00:00 2001 From: Rifa Achrinza <25147899+achrinza@users.noreply.github.com> Date: Mon, 11 Nov 2024 01:01:04 +0800 Subject: [PATCH] feat: ci/cd commmon scripts and docs Signed-off-by: Rifa Achrinza <25147899+achrinza@users.noreply.github.com> --- .gitignore | 5 +++ docs/cicd-scripts.md | 74 +++++++++++++++++++++++++++++++++++ docs/cicd-services.md | 70 +++++++++++++++++++++++++++++++++ shared-scripts/cicd/common.sh | 20 ++++++++++ 4 files changed, 169 insertions(+) create mode 100644 docs/cicd-scripts.md create mode 100644 docs/cicd-services.md create mode 100644 shared-scripts/cicd/common.sh diff --git a/.gitignore b/.gitignore index dc81e496..6ace3068 100644 --- a/.gitignore +++ b/.gitignore @@ -105,3 +105,8 @@ dist # TernJS port file .tern-port + +# Code editors +\#*# +.#* +*~ diff --git a/docs/cicd-scripts.md b/docs/cicd-scripts.md new file mode 100644 index 00000000..83160681 --- /dev/null +++ b/docs/cicd-scripts.md @@ -0,0 +1,74 @@ +# CI/CD Scripts + +This document defines the file locations and purpose of: + +1. CI/CD scripts within each Git repository +2. Shareable CI/CD scripts (to be `source`d) in the `loopback/cicd` Git repository + +## Repository-Specific CI/CD Scripts + +### Rationale + + + + + + + + + + + + + + + + + + + + + + + + + +
RationaleDescription
FlexibilityNPM Scripts are useful for one-liners. However, test harnesses are typically more complex, and NPM Scripts designed for test harnesses may not be ideal for developer experience. Discrete CI/CD scripts provide the flexibility and independence needed to accomodate the bespoke requirements of CI/CD pipelines.
Cross-testing + The LoopBack project has NPM packages that span across several Git repositories. Some of these packages are dependents or dependencies of each other. Hence, there is a need to support cross-testing with each other.
+ For example, the dependency chain `loopback-connector-db2`->`loopback-ibmdb`->`loopback-datasource-juggler` means that making a change in one of these packages may break the other packages. Cross-testing is manatory to reduce the likelihood of breakage.
+ Furthermore, we have to test both directions of the dependency chain, "upstream" (dependencies) and "downstream" (dependents), and we must test across supported major versions due to the adoption of the Module LTS policy. +
CI-agnostismLoopBack uses a [mixture of CI/CD services](./cicd-services.md). This is means that usage of CI/CD service-specific configuration should be kept to a minimum to reduce redundant authoring.
Ease of Bootstrap + Performing certain actions as part of CI/CD actions (e.g. testing) may require setup that's unique for that Git Repository.
+ The purpose of repo-specific CI/CD scripts is to perform well-defined actions in a single line. +
+ +### File Locations + +| Location | Remarks | +|------------------------------------|-----------------------------------------------------------------------------------------------| +| `/cicd/*` | All CI/CD resources (scripts, submodules, supporting files) MUST be stored in this directory | +| `/cicd/well-known/*.sh` | MUST contain only scripts defined in this document | +| `/cicd/well-known/*-supplements/*` | Internal supplementary files for well-known scripts | +| `/cicd/well-known/set-env/*.sh` | MUST contain only scripts defined in this document. Intended to be `source`d to set env vars. | +| `/cicd/vendor/*` | Internal Git submodules for well-known scripts | + +| Well-Known Scripts | Input Env Vars | Required? | Remarks | +|-----------------------------|-----------------------------|-----------|----------------------------------------------------------------| +| `prepare-autoinstall.sh` | | Yes | | +| `test-harness-setup-env.sh` | `CI_NODEJS_AUTOINSTALL_DIR` | Yes | Setup environment for test harness (database, `npm ci`, etc.) | +| `test-harness.sh` | | Yes | MUST NOT be execued before `test-harness-setup-env.sh` script. | +| `test-harness-cleanup.sh` | | No | Cleanup environment (e.g. database containers) | + +| Well-Known `set-env` Scripts | Output Env Vars | Required? | Remarks | +|-------------------------------|-----------------------------|-----------|------------------------------------------------------------| +| `post-prepare-autoinstall.sh` | `CI_NODEJS_AUTOINSTALL_DIR` | Yes | MUST NOT be execued before `prepare-autoinstall.sh` script | + +| Env Vars | Remarks | Producer | Consumer | +|-----------------------------|---------------------------------------------------------------------------------------|--------------------------|----------| +| `CI_NODEJS_AUTOINSTALL_DIR` | `PATH`-style list of directories contianing `.tgz` archives generated from `npm pack` | `prepare-autoinstall.sh` | | + +## Shareable CI/CD Scripts + +| Script | Internal Fn/Var Prefix | Public Fn/Var | +|-------------|------------------------|---------------------------------------------| +| `common.sh` | `__CI_COMMON` | `ORIG_DIR`, `BASE_DIR`, `step`, `mktempdir` | diff --git a/docs/cicd-services.md b/docs/cicd-services.md new file mode 100644 index 00000000..6d98f9a4 --- /dev/null +++ b/docs/cicd-services.md @@ -0,0 +1,70 @@ +# CI/CD Services + +This document provides informtaion on the CI/CD services available to the LoopBack project. + +## CI/CD Service Matrix + +These matrixes act as a rough guideline on which CI/CD service is used with what configuration, and rationale for unused configurations. + +### Actively Used + +| CI | CPU Arch | OS | Docker? | Env | SSH Access | Remarks | +|----------------|----------|---------|----------|-----------------|------------|------------------------------------------------------------------------------------| +| GitHub Actions | x86-64 | Ubuntu | Yes | Virtual machine | No | | +| GitHub Actions | x86-64 | macOS | No | Virtual machine | No | No Docker support OOTB (manual install too slow), hence not used by ORM connectors | +| AppVeyor | x86-64 | Windows | Yes | Virtual machine | Unknown | | +| Travis CI | arm | Ubuntu | Limited^ | LXD | No | | +| Travis CI | ppc64le | Ubuntu | Limited^ | LXD | No | | +| Travis CI | s390x | Ubuntu | Limited^ | LXD | No | | + +^LXD containers have stricter limitations such as smaller SUID/GUID pool + +### Available but Unused + +| CI/CD Service | CPU Arch | OS | Remarks | +|----------------|----------|---------|------------------------------------------------------------| +| GitHub Actions | x86-64 | Windows | No Docker support | +| GitHub Actions | arm | Ubuntu | Not available for FOSS yet | +| GitHub Actions | x86-64 | macOS | | +| GitHub Actions | arm | macOS | No Docker support (No nested virtualisation in M1) | +| Travis CI | x86-64 | Any | Uses credits, which requires Customer Support to replenish | + +## Security + +### Secrets Management + +No long-term secrets are stored in the CI/CD services. However, there is no service to continously prove this as true at this moment. + +GitHub Actions exposes `GITHUB_TOKEN` which is restricted with top-level `permisions: {}`. Permissions are added on a as-needed basis. Known cases where elevated permissions are given: + +- GitHub CodeQL +- Pipeline publishing (Not ready yet) + +### Security Against Untrusted PRs and Forks + +Security features: + +| Feature \ CI/CD Service | GitHub Actions | Travis CI (excl. dpl) | AppVeyor | Remarks | +|-------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|----------|----------------------------------------------------------------| +| Require Approval for outside contributors | Yes | No | No | | +| No secrets exposed on fork runs | [Yes (enforced)](https://github.com/github/docs/blob/bc905e2b1be277e49445aa6ba80a2afc04b2cfc2/content/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions.md#potential-impact-of-a-compromised-runner) | [Yes](https://changelog.travis-ci.com/git-repository-security-settings-for-forks-224005) | ? | | +| Ephemeral token | Yes (manual) (`GITHUB_TOKEN`) | No | ? | Enable verification that a request came from a CI pipeline run | +| Ephemeral GitHub token | Yes (enforced) | N/A | ? | | +| Restricted GitHub token | Yes (manual) | N/A | ? | | +| Network Restriction | Yes (manual) ([Step Security Harden Runner](https://github.com/step-security/harden-runner)) | No | No | Mitigate data exfiltration | + +Legend: +- **?:** Unknown. +- **N/A:** Not applicable to that CI/CD platform; Hence no additional security risk. +- **No:** No such security mechanism available on CI/CD platform. +- **Yes (enforced):** Enforced by the the CI/CD platform without additional configuration. +- **Yes (manual):** Enforced through manual, non-centralised configuration. +- **Yes (centralised):** Enforced through centralised configuration via `loopback/cicd` CI pipelines. + +Based on the feature list above, CI/CD services do not provide sufficient security boundaries against secret exfiltration. + +### Recommendations + +- **Long-term secrets MUST NOT be made available to CI/CD pipeline environments** +- **For non-critical services, use ephemeral tokens for non-critical services** that SHOULD only last for the duration of the CI/CD pipeline run or expire quickly. +- **For critical services (e.g. publishing), use short-lived tokens with out-of-band 2FA** to ensure that an additional, out-of-pipeline authentication factor is required. diff --git a/shared-scripts/cicd/common.sh b/shared-scripts/cicd/common.sh new file mode 100644 index 00000000..4300beea --- /dev/null +++ b/shared-scripts/cicd/common.sh @@ -0,0 +1,20 @@ +ORIG_DIR="$(pwd)" +cd "$(dirname "$0")/../.." +BASE_DIR="$(pwd)" + +CI_NODEJS_AUTOINSTALL_DIR="$BASE_DIR/cicd/tmp/nodejs-autoinstall" +PREPARE_POSTINSTALL_SCRIPT="$BASE_DIR/cicd/tmp/well-known/set-env/post-prepare-autoinstall.sh" + +__CI_COMMON_STEP_COUNT=1 + +step () { + printf "\n\n============================================================================\n" + printf 'STEP #%d: %s\n' "$STEP_COUNT" "$1" + printf "\n============================================================================\n\n" + _CI_COMMON_STEP_COUNT="$((STEP_COUNT + 1))" +} + +mktempdir () { + # TODO: Provide a POSIX-compliant fallback + printf '%s' "$(mktemp -d)" +}