Merge pull request #188 from GeoscienceAustralia/reformatting

abradley60 · web-flow · commit af4c27d40192 · 2026-05-05T11:34:01.000+09:30
Reformatting
diff --git a/README.md b/README.md
@@ -34,13 +34,13 @@ standards‑compliant distribution and downstream use.
 1. Build the container
 
 ```bash
-docker build -t sar-pipeline -f Docker/isce3_rtc/Dockerfile .
+docker build -t sar-pipeline-isce3-rtc -f Docker/isce3_rtc/Dockerfile .
 ```
 
 2. Test the image interactively (type `exit` to exit)
 
 ```bash
-docker run -it --entrypoint /bin/bash sar-pipeline
+docker run -it --entrypoint /bin/bash sar-pipeline-isce3-rtc
 ```
 
 1. Set the following minimum environment credentials in a `.env` file. At minimum we require earthdata *OR* Coperniucs Space Data Ecosystem (CDSE) credentials to download from the Alaska Satelite Facility (ASF) or CDSE respectively. These can be created here for the [ASF](http://urs.earthdata.nasa.gov/) and [CDSE](https://dataspace.copernicus.eu/).
@@ -57,13 +57,13 @@ mkdir data
 ```
 
 ```bash
-docker run --env-file .env -v ${PWD}/data:/home/rtc_user/working sar-pipeline \
+docker run --env-file .env -v ${PWD}/data:/home/rtc_user/working sar-pipeline-isce3-rtc \
 --scene S1A_IW_SLC__1SSH_20220101T124744_20220101T124814_041267_04E7A2_1DAD \
---burst_id_list t070_149815_iw3 \
---skip_upload_to_s3 \
---make_existing_products \
---scene_data_source ASF \
---orbit_data_source ASF 
+--burst-id-list t070_149815_iw3 \
+--skip-upload-to-s3 \
+--make-existing-products \
+--scene-data-source ASF \
+--orbit-data-source ASF 
 ```
 
 Note if there are permission issues writing to the local `./data` folder, the following can be run:
@@ -95,6 +95,10 @@ sudo chmod -R 777 ./data
 
 6. See the [full docs](docs/pipelines/isce3_rtc.md) to see how static layers can created and used.
 
+## Release
+
+GA release information is provided in the [release guide](./docs/development/release_guide.md)
+
 ## License
 
 Copyright © 2025 Geoscience Australia.
diff --git a/docs/development/release_guide.md b/docs/development/release_guide.md
@@ -1,21 +1,47 @@
-# AWS Opera/iSCE3 Pipeline
+# ISCE3-RTC Pipeline (s1-nrb-pipeline)
 
 ### Steps
 
 1. Branch off from main to develop new features for a planned release.
 2. Run the end-to-end tests locally on the branch with proposed changes. Note the local run requires sufficient compute and credentials to be set. 
 
 ```bash
-# test command line tools
-pixi run test-aws-cli 
-
-# test docker image build and test run
-pixi run test-full-aws-docker-run
+# tests that don't require credentials
+pixi run test-pipeline-no-creds
+# tests to download data from sources. Requires credentials in .env
+pixi run test-scene-data-source-queries
+# full test of isce3 rtc pipeline. Note may take a few hours as the docker image is
+# built and run for a few products. Requires credentials in .env
+pixi run test-isce3-rtc
 ```
+
 3. If successful raise a PR for review and merge branch into main
 4. Create new release on GitHub and increment the tag: 
       - Images prior to an initial release should be of for, `v0.X.X`.
       - Test images should be of form `vX.X.X_betaX` 
       - For small changes the tag should be incremented by `0.0.1`. For example, `v1.0.1` -> `v1.0.2`. 
       - Major changes should increment the first or second number, depending on the impact of the change. 
-5. When the release made, the `push-image-to-ecr.yaml` github action will be run. A new image will be built, tagged and pushed to ECR according to the release tag. E.g. `sar_pipeline:v1.0,1`
+5. The release should trigger the workflow [push-image-to-ecr](../../.github/workflows/push-image-to-ecr.yaml) that will build and push the updated image to the AWS ECR repository.
+6. If the automated build and push fails, manually tag and upload the docker image to the ECR repository:
+
+```bash
+# set credentials with write to the ECR repo, or use access tool 
+# like granted / assume to set credentials
+export AWS_ACCESS_KEY_ID=
+export AWS_SECRET_ACCESS_KEY=
+export AWS_SESSION_TOKEN=
+
+docker build -t sar-pipeline-isce3-rtc -f Docker/isce3_rtc/Dockerfile .
+
+# NOTE - following instructions are for dev account - 451924316694
+
+# tag the local image with the appropriate ECR account tag
+docker tag sar-pipeline-isce3-rtc:vX-X-X 451924316694.dkr.ecr.ap-southeast-2.amazonaws.com/dea-dev-s1-nrb-pipeline:vX-X-X
+
+aws ecr get-login-password \
+    --region ap-southeast-2 | docker login \
+    --username AWS \
+    --password-stdin 451924316694.dkr.ecr.ap-southeast-2.amazonaws.com
+
+docker push 451924316694.dkr.ecr.ap-southeast-2.amazonaws.com/dea-dev-s1-nrb-pipeline:vX.X.X
+```
diff --git a/docs/pipelines/isce3_rtc.md b/docs/pipelines/isce3_rtc.md
diff --git a/environment.yaml b/environment.yaml
@@ -20,6 +20,7 @@ dependencies:
 - tomli >=2.2.1
 - s1reader >=0.2.4
 - aioboto3 ==14.1.0
+- s3fs >=2026.3.0,<2027
 - python >=3.8
 - pip
 - pip:
@@ -37,5 +38,9 @@ dependencies:
   - dem-handler @ git+https://github.com/GeoscienceAustralia/dem-handler.git@v0.2.6
   - odc-stac>=0.5.0, <0.6
   - pystac-client>=0.9.0, <0.10
-  - tqdm>=4.67.1, <5
   - sentineleof @ git+https://github.com/scottstanie/sentineleof@v0.11.2
+  - tqdm>=4.67.1, <5
+  - black[jupyter]==26.3.1
+  - ipykernel>=7.2.0, <8
+  - stac-geoparquet>=0.7.0, <0.8
+  - docker>=7.1.0, <8
diff --git a/pixi.lock b/pixi.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -32,7 +32,7 @@ dependencies = [
     "pystac-client>=0.9.0,<0.10", 
     "sentineleof @ git+https://github.com/scottstanie/sentineleof@v0.11.2",
     "tqdm>=4.67.1,<5", 
-    "black[jupyter]>=26.3.1,<27", 
+    "black[jupyter]==26.3.1", 
     "ipykernel>=7.2.0,<8", 
     "stac-geoparquet>=0.7.0,<0.8",
     "docker>=7.1.0,<8"
diff --git a/scripts/run_isce3_rtc_pipeline.sh b/scripts/run_isce3_rtc_pipeline.sh
@@ -44,22 +44,22 @@ while [[ "$#" -gt 0 ]]; do
     case $1 in
         --scene) scene="$2"; shift 2 ;;
         --resolution) resolution="$2"; shift 2 ;;
-        --output_crs) output_crs="$2"; shift 2 ;;
-        --dem_type) dem_type="$2"; shift 2 ;;
+        --output-crs) output_crs="$2"; shift 2 ;;
+        --dem-type) dem_type="$2"; shift 2 ;;
         --product) product="$2"; shift 2 ;;
-        --backscatter_convention) backscatter_convention="$2"; shift 2 ;;
-        --static_layer_validity_start_date) static_layer_validity_start_date="$2"; shift 2 ;;
-        --s3_bucket) s3_bucket="$2"; shift 2 ;;
-        --s3_project_folder) s3_project_folder="$2"; shift 2 ;;
-        --collection_number) collection_number="$2"; shift 2 ;;
-        --make_existing_products) make_existing_products=true; shift ;;
-        --skip_upload_to_s3) skip_upload_to_s3=true; shift ;;
-        --link_static_layers) link_static_layers=true; shift ;;
-        --linked_static_layers_s3_bucket) linked_static_layers_s3_bucket="$2"; shift 2 ;;
-        --linked_static_layers_collection_number) linked_static_layers_collection_number="$2"; shift 2 ;;
-        --linked_static_layers_s3_project_folder) linked_static_layers_s3_project_folder="$2"; shift 2 ;;
-        --processed_scene_tracking_file_s3_folder) processed_scene_tracking_file_s3_folder="$2"; shift 2 ;;
-        --scene_data_source) 
+        --backscatter-convention) backscatter_convention="$2"; shift 2 ;;
+        --static-layer-validity-start-date) static_layer_validity_start_date="$2"; shift 2 ;;
+        --s3-bucket) s3_bucket="$2"; shift 2 ;;
+        --s3-project-folder) s3_project_folder="$2"; shift 2 ;;
+        --collection-number) collection_number="$2"; shift 2 ;;
+        --make-existing-products) make_existing_products=true; shift ;;
+        --skip-upload-to-s3) skip_upload_to_s3=true; shift ;;
+        --link-static-layers) link_static_layers=true; shift ;;
+        --linked-static-layers-s3-bucket) linked_static_layers_s3_bucket="$2"; shift 2 ;;
+        --linked-static-layers-collection-number) linked_static_layers_collection_number="$2"; shift 2 ;;
+        --linked-static-layers-s3-project-folder) linked_static_layers_s3_project_folder="$2"; shift 2 ;;
+        --processed-scene-tracking-file-s3-folder) processed_scene_tracking_file_s3_folder="$2"; shift 2 ;;
+        --scene-data-source) 
             scene_data_source=()  # clear the default
             shift
             if [[ $# -eq 1 && -f "$1" ]]; then
@@ -72,7 +72,7 @@ while [[ "$#" -gt 0 ]]; do
                 done
             fi
             ;;
-        --orbit_data_source)
+        --orbit-data-source)
             orbit_data_source=()  # clear the default
             shift
             if [[ $# -eq 1 && -f "$1" ]]; then
@@ -85,9 +85,9 @@ while [[ "$#" -gt 0 ]]; do
                 done
             fi
             ;;
-        --skip_validate_stac) skip_validate_stac=true; shift ;;
-        --skip_rtc) skip_rtc=true; shift ;;
-        --burst_id_list)
+        --skip-validate-stac) skip_validate_stac=true; shift ;;
+        --skip-rtc) skip_rtc=true; shift ;;
+        --burst-id-list)
             shift
             if [[ $# -eq 1 && -f "$1" ]]; then
                 burst_id_list=($(cat "$1"))
@@ -125,7 +125,7 @@ if [[ -z "$output_crs" || "${output_crs,,}" == "utm" ]]; then
 elif [[ "$output_crs" =~ ^[0-9]+$ ]]; then
     epsg_code_msg="EPSG:$output_crs"
 else
-    echo "Error: --output_crs must be empty, 'UTM', 'utm', or an integer corresponding to an EPSG code (e.g. 3031)."
+    echo "Error: --output-crs must be empty, 'UTM', 'utm', or an integer corresponding to an EPSG code (e.g. 3031)."
     exit 1
 fi
 
@@ -139,29 +139,29 @@ fi
 echo ""
 echo The input variables are:
 echo scene : "$scene"
-echo burst_id_list : ${burst_id_list[*]}
+echo burst-id-list : ${burst_id_list[*]}
 echo resolution : "$resolution"
-echo output_crs : "$epsg_code_msg"
-echo dem_type :  "$dem_type"
+echo output-crs : "$epsg_code_msg"
+echo dem-type :  "$dem_type"
 echo product : "$product"
-echo backscatter_convention : "$backscatter_convention"
-echo static_layer_validity_start_date : "$static_layer_validity_start_date"
-echo s3_bucket : "$s3_bucket"
-echo s3_project_folder : "$s3_project_folder"
-echo collection_number : "$collection_number"
-echo make_existing_products : "$make_existing_products"
-echo skip_upload_to_s3 : "$skip_upload_to_s3"
-echo scene_data_source : ${scene_data_source[*]}
-echo orbit_data_source : ${orbit_data_source[*]}
-echo skip_validate_stac : "$skip_validate_stac"
-echo processed_scene_tracking_file_s3_folder : "$processed_scene_tracking_file_s3_folder"
-echo skip_rtc : "$skip_rtc"
+echo backscatter-convention : "$backscatter_convention"
+echo static-layer-validity-start-date : "$static_layer_validity_start_date"
+echo s3-bucket : "$s3_bucket"
+echo s3-project-folder : "$s3_project_folder"
+echo collection-number : "$collection_number"
+echo make-existing-products : "$make_existing_products"
+echo skip-upload-to-s3 : "$skip_upload_to_s3"
+echo scene-data-source : ${scene_data_source[*]}
+echo orbit-data-source : ${orbit_data_source[*]}
+echo skip-validate-stac : "$skip_validate_stac"
+echo processed-scene-tracking-file-s3-folder : "$processed_scene_tracking_file_s3_folder"
+echo skip-rtc : "$skip_rtc"
 
 # warn the user about linking static layers
 if [[ "$link_static_layers" = true && "$product" = "RTC_S1" ]]; then
-    echo linked_static_layers_s3_bucket : "$linked_static_layers_s3_bucket"
-    echo linked_static_layers_collection_number : "$linked_static_layers_collection_number"
-    echo linked_static_layers_s3_project_folder : "$linked_static_layers_s3_project_folder"
+    echo linked-static-layers-s3-bucket : "$linked_static_layers_s3_bucket"
+    echo linked-static-layers-collection-number : "$linked_static_layers_collection_number"
+    echo linked-static-layers-s3-project-folder : "$linked_static_layers_s3_project_folder"
     echo ""
     echo 'WARNING: RTC_S1_STATIC layers are being linked to the RTC_S1 products in STAC metadata'
     echo 'For more information, see the workflow documentation'
@@ -176,9 +176,9 @@ scratch_folder="/home/rtc_user/working/scratch/$s3_project_folder/$collection_nu
 
 echo ""
 echo The container will use these paths for processing:
-echo download_folder : "$out_folder"
-echo scratch_folder : "$scratch_folder"
-echo out_folder : "$out_folder"
+echo download-folder : "$out_folder"
+echo scratch-folder : "$scratch_folder"
+echo out-folder : "$out_folder"
 echo ""
 
 # activate conda 
diff --git a/tests/sar_pipeline/isce3_rtc/test_full_docker_workflow_run.py b/tests/sar_pipeline/isce3_rtc/test_full_docker_workflow_run.py
@@ -66,7 +66,7 @@
 
 # shared test values
 DOCKER_TAG = re.sub(r"[^a-zA-Z0-9_.-]", "-", sar_pipeline.__version__)
-# DOCKER_TAG = '0.4.1.dev36-g6eecb22a7.d20250925' # set existing image
+# DOCKER_TAG = '0.7.2.dev23-g70bf0da26' # set existing image
 RUN_DATETIME = str(datetime.now()).replace(" ", "_").replace(":", "-")
 # RUN_DATETIME = '2025-12-04_05-46-40.829598'  # set existing datetime
 TEST_NAME = Path(__file__).stem
@@ -147,14 +147,14 @@
 def build_image():
     """build and tag the docker image for the current codebase to be used in tests"""
     logging.info(
-        f"Building docker image sar-pipeline:{DOCKER_TAG} for testing, this may take a few minutes..."
+        f"Building docker image sar-pipeline-isce3-rtc:{DOCKER_TAG} for testing, this may take a few minutes..."
     )
     result = subprocess.run(
         [
             "docker",
             "build",
             "-t",
-            f"sar-pipeline:{DOCKER_TAG}",
+            f"sar-pipeline-isce3-rtc:{DOCKER_TAG}",
             "-f",
             "Docker/isce3_rtc/Dockerfile",
             ".",
@@ -186,7 +186,7 @@ def _run_docker_for_scene(
     Parameters
     ----------
     docker_image_tag : str
-        the docker image tag for sar-pipeline. e.g.
+        the docker image tag for sar-pipeline-isce3-rtc. e.g.
          0.4.1.dev256-g9b7a3f1ea.d20251204
     docker_env_list : list
         A list of environment variables to be passed
@@ -218,12 +218,12 @@ def _run_docker_for_scene(
 
     # mount the local directory for results
     volume_mount_list = ["-v", f"{local_outputs_folder}:/home/rtc_user/working/results"]
-    # Optional -> mount somewhere to store downloads
+    # # Optional -> mount somewhere to store downloads
     # volume_mount_list += [
     #     "-v",
     #     f"/data/working/downloads:/home/rtc_user/working/downloads",
     # ]
-    # Optional -> mount the scratch directory
+    # # Optional -> mount the scratch directory
     # volume_mount_list += ["-v", f"/data/working/scratch:/home/rtc_user/working/scratch"]
 
     cmd = [
@@ -232,20 +232,20 @@ def _run_docker_for_scene(
         *volume_mount_list,
         "--rm",
         *docker_env_list,
-        f"sar-pipeline:{docker_image_tag}",
+        f"sar-pipeline-isce3-rtc:{docker_image_tag}",
         "--scene",
         scene,
-        "--burst_id_list",
+        "--burst-id-list",
         burst_id,
         "--product",
         "RTC_S1_STATIC",
-        "--backscatter_convention",
+        "--backscatter-convention",
         "gamma0",
-        "--collection_number",
+        "--collection-number",
         "1",
-        "--s3_bucket",
+        "--s3-bucket",
         s3_bucket,
-        "--s3_project_folder",
+        "--s3-project-folder",
         s3_project_folder,
     ]
     result = subprocess.run(
@@ -265,27 +265,27 @@ def _run_docker_for_scene(
         *volume_mount_list,
         "--rm",
         *docker_env_list,
-        f"sar-pipeline:{docker_image_tag}",
+        f"sar-pipeline-isce3-rtc:{docker_image_tag}",
         "--scene",
         scene,
-        "--burst_id_list",
+        "--burst-id-list",
         burst_id,
         "--product",
         "RTC_S1",
-        "--backscatter_convention",
+        "--backscatter-convention",
         "gamma0",
-        "--collection_number",
+        "--collection-number",
         "1",
-        "--s3_bucket",
+        "--s3-bucket",
         s3_bucket,
-        "--s3_project_folder",
+        "--s3-project-folder",
         s3_project_folder,
-        "--link_static_layers",
-        "--linked_static_layers_s3_bucket",
+        "--link-static-layers",
+        "--linked-static-layers-s3-bucket",
         s3_bucket,
-        "--linked_static_layers_s3_project_folder",
+        "--linked-static-layers-s3-project-folder",
         s3_project_folder,
-        "--linked_static_layers_collection_number",
+        "--linked-static-layers-collection-number",
         "1",
     ]
 
