Updates/compare products

.dockerignore

@@ -2,4 +2,4 @@
 .pytest_cache
 *.secret
 *.env
-tests/sar_pipeline/data/isce3_rtc/TMP/*
+tests/sar_pipeline/isce3_rtc/data/TMP/*

.gitignore

@@ -44,3 +44,6 @@ sar_pipeline/_version.py
 
 # ignore created test files
 tests/sar_pipeline/isce3_rtc/data/TMP/*
+
+tests/sar_pipeline/data/isce3_rtc/TMP/*
+comparison

Conda/pygssearch/environment.yaml

@@ -2,7 +2,7 @@ name: pygssearch-env
 channels:
   - conda-forge
 dependencies:
-  - python>=3.8
+  - python=3.10
   - pip
   - pip:
       - pygssearch==1.2.4

README.md

@@ -1,17 +1,72 @@
 # sar-pipeline
 
-This repository contains code for running SAR processing pipelines on the NCI and AWS.
+This repository contains code for running SAR processing pipelines on the NCI and AWS. Currently, this codebase supports two pipelines for generating Sentinel-1 Normalised Radar Backscatter (NRB). 
 
-## Environment set up
+* [isce3_rtc (Sentinel-1 IW) that can be run locally and on AWS](docs/pipelines/aws_isce3_rtc.md)
+* [pyroSAR-GAMMA (Sentinel-1 IW/EW) that can be run on the NCI](docs/pipelines/pyrosar_gamma.md)
 
-For more information see [Setup](docs/development/README.md)
+For more information see [Pipelines](docs/pipelines/README.md) or the specific workflow docs for usage examples and running tests.
+
+## Development Setup
+
+Detailed documentation for the project setup can be found in the [development documentation](docs/development/README.md). It is highly recommended this be reviewed before contributing to the project. This project utilises [pixi](https://pixi.sh/latest/) for managing packages and running tests.
 
 * [Cloning the project](docs/development/README.md)
 * [Developer set up](docs/development/developer_pixi.md)
 * [User set up](docs/development/user_conda.md)
 
-## Workflows
+## Quick Setup
+
+Clone the repository
+
+```bash
+git clone https://github.com/GeoscienceAustralia/sar-pipeline.git
+```
+
+### ISCE3 RTC (Docker)
+
+1. Build the container
+
+```bash
+docker build -t sar-pipeline -f Docker/isce3_rtc/Dockerfile .
+```
+
+2. Test the image interactively (type `exit` to exit)
+
+```bash
+docker run -it --entrypoint /bin/bash sar-pipeline
+```
+
+1. Set the following minimum environment credentials in a `.env` file. At minimum we require AWS credentials and a set of credentials to download data. Earthdata credentials to download from the Alaska Satelite Facility (ASF) can be created [here](http://urs.earthdata.nasa.gov/).
+
+```text
+EARTHDATA_LOGIN=
+EARTHDATA_PASSWORD=
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+AWS_DEFAULT_REGION="ap-southeast-2"
+```
+
+4. Generate backscatter data for a test burst. The outputs will be written to a local `data` folder.
+
+```bash
+mkdir data
+```
+
+```bash
+docker run --env-file .env -v ${PWD}/data:/home/rtc_user/working sar-pipeline:0.4.1.dev36-g6eecb22a7.d20250925  \
+--scene S1A_IW_SLC__1SSH_20220101T124744_20220101T124814_041267_04E7A2_1DAD \
+--burst_id_list t070_149815_iw3 \
+--skip_upload_to_s3 \
+--make_existing_products \
+--scene_data_source ASF \
+--orbit_data_source ASF 
+```
+
+Note if there are permission issues writing to the local `./data` folder, the can be run.
+
+```bash
+sudo chmod -R 777 ./data
+```
+
 
-For more information see [Workflows](docs/pipelines/README.md)
-* [NCI (pyroSAR+GAMMA)](docs/pipelines/pyrosar_gamma.md)
-* [AWS (ISCE3_RTC)](docs/pipelines/aws_isce3_rtc.md)

docs/pipelines/README.md

@@ -1,49 +1,8 @@
-# Workflows
-The repository will handle multiple workflows and backends, some of which have specific requirements.
+# Pipelines
+The repository will handle multiple pipelines and backends, some of which have specific requirements.
 
 ## Workflows
 - pyroSAR + GAMMA
     - [Requirements](pyrosar_gamma.md)
-
-## Command line interface
-> **_NOTE:_**  At this time, pyroSAR + GAMMA is the only workflow, and the command line interfaces have not yet been generalised to other workflows.
-
-The package has a number of useful command line interfaces:
-
-### Finding the location of a scene on the NCI
-The `find-scene` command will display the location of a given scene on the NCI.
-The full path to the scene is required as the input to other commands.
-
-Example usage:
-```
-$ find-scene S1A_EW_GRDM_1SDH_20240129T091735_20240129T091828_052319_065379_0F1E
-
-/path/to/scene/S1A_EW_GRDM_1SDH_20240129T091735_20240129T091828_052319_065379_0F1E.zip
-```
-
-### Submit a workflow
-This will submit a job request to the NCI based on the job parameters and file paths in the supplied config. 
-The [default config](../../sar_pipeline/nci/configs/default.toml) will be used if no other config is provided.
-
-Example usage
-```
-$ submit-pyrosar-gamma-workflow /path/to/scene/S1A_EW_GRDM_1SDH_20240129T091735_20240129T091828_052319_065379_0F1E.zip
-```
-This will submit a job to the NCI with the default config.
-To use a different config, run the command and supply the `--config` option
-```
---config /path/to/config.toml
-```
-
-### Run a workflow interactively
-If you are still testing a workflow, it is best to run it in an interactive session.
-While in an interactive session, you can run the workflow directly. 
-
-Example usage
-```
-$ run-pyrosar-gamma-workflow /path/to/scene/S1A_EW_GRDM_1SDH_20240129T091735_20240129T091828_052319_065379_0F1E.zip
-```
-To use a different config, run the command and supply the `--config` option
-```
---config /path/to/config.toml
-```
+- isce3_rtc
+    - [Requirements](isce3_rtc.md)