Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# GitHub syntax highlighting
pixi.lock linguist-language=YAML linguist-generated=true
4 changes: 4 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,7 @@ updates:
directory: "/"
schedule:
interval: "weekly"
groups:
dependencies:
patterns:
- "*"
52 changes: 27 additions & 25 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,39 +18,41 @@ concurrency:

defaults:
run:
# See https://github.com/marketplace/actions/setup-micromamba#about-login-shells
shell: bash -leo pipefail {0}

jobs:
test:
unit-tests:
runs-on: ubuntu-latest
strategy:
fail-fast: false

steps:
- name: Checkout source files
uses: actions/checkout@v4

- name: Setup micromamba and install/cache dependencies
env:
# Prevent timeouts with micromamba (libmamba) during dependency installation
MAMBA_NO_LOW_SPEED_LIMIT: 1
uses: mamba-org/setup-micromamba@v2
with:
cache-environment: true
create-args: --category main dev
environment-file: conda-lock.yml
environment-name: gedi_subset
micromamba-version: 1.5.10-0

- name: Install GEDI Subsetter
run: python -m pip install --no-deps -e .
- name: Setup Pixi
uses: prefix-dev/setup-pixi@v0.8.3

- name: Run unit tests
run: |
make test
run: pixi run test

type-analysis:
runs-on: ubuntu-latest
steps:
- name: Checkout source files
uses: actions/checkout@v4

- name: Setup Pixi
uses: prefix-dev/setup-pixi@v0.8.3

- name: Run static type analysis
run: pixi run mypy

validate-installation:
runs-on: ubuntu-latest
steps:
- name: Checkout source files
uses: actions/checkout@v4

- name: Setup Pixi
uses: prefix-dev/setup-pixi@v0.8.3

- name: Run type checks
if: always()
run: |
make typecheck
- name: Validate "prod" installation
run: bin/build.sh
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.pixi
.envrc
maap.cfg
memray-*
Expand Down
3 changes: 1 addition & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ repos:
rev: v5.0.0
hooks:
- id: check-added-large-files
exclude: conda-lock.yml
exclude: pixi.lock
- id: check-case-conflict
- id: check-executables-have-shebangs
- id: check-json
Expand All @@ -37,7 +37,6 @@ repos:
# optional: list of Conventional Commits types to allow e.g. [feat, fix, ci, chore, test]
args: []
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.9.7
hooks:
- id: ruff
Expand Down
94 changes: 25 additions & 69 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Contributing

- [Development Setup](#development-setup)
- [Managing Dependencies](#managing-dependencies)
- [Testing](#testing)
- [Testing CMR Queries](#testing-cmr-queries)
- [Linting and Running Unit Tests](#linting-and-running-unit-tests)
Expand All @@ -16,67 +15,26 @@ To contribute to this work, you must obtain access to the [NASA MAAP], where the
Algorithm Development Environment (ADE) resides, and thus where algorithms can
be registered and launched.

If you wish to conduct development work outside of the ADE, you'll need to have
the following installed in your environment (the ADE has these pre-installed):
You'll need to have the following installed in your development environment
(wherever you plan to conduct development, within the ADE or not):

- `git`: On macOS, using Homebrew is highly recommended: `brew install git`.
Otherwise, see <https://git-scm.com/downloads>.
- `conda`: On macOS, using Homebrew to install Miniforge (containing `conda`) is
highly recommended: `brew install --cask miniforge`. Otherwise, see
<https://github.com/conda-forge/miniforge>.
- `git` (already installed in the ADE): On macOS, using Homebrew is highly
recommended: `brew install git`. Otherwise, see
<https://git-scm.com/downloads>.
- `pixi`: See [Pixi Installation](https://pixi.sh/latest/#installation)
(**Hint:** if you want to use the `curl` command, but don't have `curl`
installed, but you have `wget` installed, you can replace `curl -fsSL` with
`wget -qO-` [note the capital letter `O` and the trailing dash])

You must also have `make` installed. On macOS, `make` should already be
installed. On Linux, if not already installed, you must locate installation
instructions appropriate for your system package manager.

**In the ADE**, `make` is not installed by default (at least not yet; see
[Install `make` in all ADE images](https://github.com/MAAP-Project/Community/issues/943)),
and must be installed as follows:

```plain
apt-get update && apt-get install make -y
```

To prepare for contributing, do the following on your development system (either
within the ADE or wherever else you're conducting development):
To prepare for contributing, do the following in your development environment:

1. Clone this GitHub repository.
1. Change directory to the cloned working directory.
1. Create the `gedi_subset` conda environment: `make build`
1. Install (and run) Git pre-commit hooks: `make lint`
1. If desired, activate the `gedi_subset` conda environment:
`conda activate gedi_subset` (**NOTE:** This is not necessary for running
any of the `make` commands, which automatically use the `gedi_subset`
environment even when the environment is not activated.)

## Managing Dependencies

To minimize the time it takes to install dependencies on the Docker image
created during algorithm registration, we leverage `conda-lock` to pre-resolve
dependencies and generate a lock file, to avoid dependency resolution during
registration. This means that we manage dependencies and update the lock file
as follows:

1. Dependencies required by the algorithm are specified in `environment.yml`.
(These are the only dependencies that will be installed in DPS.)
1. Development dependencies (for testing, static code analysis, notebooks, etc.)
are specified in `environment-dev.yml`.
1. We use `conda-lock` to generate the lock file `conda-lock.yml` from the files
above, whenever we make a change to at least one of the files.

Within `environment.yml` and `environment-dev.yml` we intentionally _avoid_
specifying _exact_ (`==`) versions to avoid resolution conflicts that might
prevent successful dependency resolution by giving the resolver enough
flexibility to satisfy all requirements. When versions are specified, they are
only to ensure we use versions with necessary features or bug fixes, and
typically use the [compatible release operator] (`~=`) to constrain only the
_major_ version of a dependency.

**IMPORTANT:** Whenever you make changes to either `environment.yml` or
`environment-dev.yml`, you must regenerate `conda-lock.yml` and install the
updated dependencies into the `gedi_subset` conda environment on your
development workstation. This is done automatically by running `make build`,
which will also update the `gedi_subset` environment as necessary.
1. Install (and run) Git pre-commit hooks: `pixi run lint`
1. If desired, activate the default Pixi environment: `pixi shell` (**NOTE:**
This is not necessary for running any of the `pixi` commands, which
automatically use the necessary environment even when the environment is not
activated.)

## Testing

Expand All @@ -86,7 +44,7 @@ Actions workflow triggered by your PR will succeed.
### Testing CMR Queries

We leverage the `vcrpy` library to record responses to HTTP/S requests. When
running _existing_ tests, these recordings (_cassettes_ in `vcrp` parlance) are
running _existing_ tests, these recordings (_cassettes_ in `vcrpy` parlance) are
replayed so that repeated test executions do _not_ make repeated requests.
Therefore, if you are not adding or modifying such tests, there is no need to
have a network connection, nor any need to run the tests within the ADE.
Expand All @@ -96,17 +54,17 @@ new tests that make CMR queries (or modifying existing ones) will not only
require a network connection in order to record live responses, but will also
require that you obtain such recordings by running the new/modified tests within
the ADE in order to have the necessary auth in play. Otherwise, the CMR queries
will either fail, or produce incorrect responses.
will either fail or produce incorrect responses.

### Linting and Running Unit Tests

Linting runs a number of checks on various files, such as making sure your code
adheres to coding conventions and that the conda lock file is in sync with the
conda environment files, among other things. To "lint" the files in the repo,
as well as run unit tests, run the following:
adheres to coding conventions, among other things. To "lint" the files in the
repo, as well as run unit tests, run the following:

```plain
make lint typecheck test
pixi run lint
pixi run test
```

If you see any errors, address them and repeat the process until there are no
Expand All @@ -120,15 +78,15 @@ GitHub Actions workflows. After installing `act`, run the following command
from the root of the repo:

```plain
act pull_request
act
```

**NOTE:** `act` uses a Docker container, so this will _NOT_ work within the ADE.
You must use `act` in an environment where Docker is installed.

The command above will initially take several minutes, but subsequent runs will
execute more quickly because only the first run must pull the `act` Docker
image.
The command above will initially take several minutes, but subsequent runs
should execute more quickly because only the first run must pull the `act`
Docker image.

## Submitting a Pull Request

Expand Down Expand Up @@ -265,8 +223,6 @@ a new release:
1. Register the new release of the algorithm as described in the previous
section.

[compatible release operator]:
https://peps.python.org/pep-0440/#compatible-release
[Keep a Changelog]:
https://keepachangelog.com/en/1.0.0/
[NASA MAAP]:
Expand Down
56 changes: 0 additions & 56 deletions Makefile

This file was deleted.

2 changes: 1 addition & 1 deletion algorithm_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ repository_url: https://github.com/MAAP-Project/gedi-subsetter.git
docker_container_url: mas.maap-project.org/root/maap-workspaces/custom_images/maap_base:v4.1.0
disk_space: 20GB
queue: maap-dps-worker-32vcpu-64gb
build_command: gedi-subsetter/bin/build-dps
build_command: gedi-subsetter/bin/build.sh
run_command: gedi-subsetter/bin/subset.sh
inputs:
config: []
Expand Down
12 changes: 6 additions & 6 deletions bin/algo/delete
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,15 @@ fi

stderr "Deleting algorithm '${algorithm_id}'..."

bin=$(dirname "$(dirname "$(readlink -f "$0")")")

message=$("${bin}/run" --quiet python -c "
read -r -d '' delete_algorithm <<EOF || true
from maap.maap import MAAP

maap = MAAP()

if not (r := maap.deleteAlgorithm('${algorithm_id}')):
print(r.json()['message'])
")
if not (r := maap.deleteAlgorithm("${algorithm_id}")):
print(r.json()["message"])
EOF

message=$(pixi run --quiet --no-progress -- python -c "${delete_algorithm}")

[[ -z "${message}" ]] || (stderr "Error: ${message}" && exit 1)
10 changes: 5 additions & 5 deletions bin/algo/describe
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ algorithm_version=$(grep "^algorithm_version:" "${yaml_file}" | sed -E 's/algori
[[ "${branch}" != "main" ]] && algorithm_version=${branch}
algorithm_id="${algorithm_name}:${algorithm_version}"

bin=$(dirname "$(dirname "$(readlink -f "$0")")")

"${bin}/run" --quiet python -c "
read -r -d '' describe_algorithm <<EOF || true
import json
import sys
import tempfile
Expand All @@ -40,11 +38,13 @@ from maap.maap import MAAP

maap = MAAP()

if r := maap.describeAlgorithm('${algorithm_id}'):
if r := maap.describeAlgorithm("${algorithm_id}"):
# This response is XML, not JSON, unfortunately, so the output is not
# convenient to read.
print(r.text)
else:
print(r.text, file=sys.stderr)
sys.exit(1)
"
EOF

pixi run --quiet --no-progress -- python -c "${describe_algorithm}"
Loading