Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- [32](https://github.com/nf-core/rangeland/pull/32) Template update for nf-core/tools v3.5.1
- [33](https://github.com/nf-core/rangeland/pull/33) Nextflow strict syntax update
- [34](https://github.com/nf-core/rangeland/pull/34) Update nf-core subworkflows
- [39](https://github.com/nf-core/rangeland/pull/39) Update to FORCE version v3.10.04

### `Fixed`

Expand Down
72 changes: 55 additions & 17 deletions bin/test.R
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ args = commandArgs(trailingOnly=TRUE)


if (length(args) != 7 && length(args) != 2) {
stop("\n Error: wrong number of parameters. Usage: \n 1st arg: workflow results directory (mosaic)
stop("\n Error: wrong number of parameters. Usage: \n 1st arg: staged workflow results directory
\n 2nd-7th args: reference rasters (*.tif) in order:
woody cover change, woody cover year of change,
herbaceous cover change, herbaceous cover year of change,
peak change, peak year of change
\nor \n
1st arg: workflow results directory (mosaic) \n 2nd arg: reference directory
1st arg: staged workflow results directory \n 2nd arg: reference directory
", call.=FALSE)
}

Expand Down Expand Up @@ -48,6 +48,50 @@ compare_direction <- function(r1, r2, threshold = 0.95) {
}
}

# function to load staged pipeline outputs without relying on upstream VRT paths
load_product_rast <- function(root_dir, product_suffix) {
tif_files <- sort(list.files(
root_dir,
pattern = paste0(product_suffix, "\\.tif$"),
recursive = TRUE,
full.names = TRUE
))

if (length(tif_files) == 0) {
stop(
paste0(
"No staged TIFFs found for product '",
product_suffix,
"' under ",
root_dir
),
call. = FALSE
)
}

if (length(tif_files) == 1) {
return(rast(tif_files))
}

merge(sprc(lapply(tif_files, rast)))
}

get_change_band <- function(r) {
if ("CHANGE" %in% names(r)) {
return(r$CHANGE)
}

r[[1]]
}

get_yoc_band <- function(r) {
if ("YEAR-OF-CHANGE" %in% names(r)) {
return(r["YEAR-OF-CHANGE"])
}

r[[3]]
}


# LOAD REFERENCE
#######################################################################
Expand Down Expand Up @@ -87,37 +131,31 @@ if (length(args) == 7 ){
# input data dir
dinp <- args[1]

fname <- dir(dinp, ".*HL_TSA_LNDLG_SMA_VBL-CAO.vrt$", full.names=TRUE)

woody_cover_rast <- rast(fname)
woody_cover_rast <- load_product_rast(dinp, "HL_TSA_LNDLG_SMA_VBL-CAO")

woody_cover_changes <- woody_cover_rast$CHANGE
woody_cover_year_of_change <- woody_cover_rast["YEAR-OF-CHANGE"]
woody_cover_changes <- get_change_band(woody_cover_rast)
woody_cover_year_of_change <- get_yoc_band(woody_cover_rast)



# HERBACEOUS COVER CHANGE (VALUE OF SEASONAL APLITUDE)
#######################################################################


fname <- dir(dinp, ".*HL_TSA_LNDLG_SMA_VSA-CAO.vrt$", full.names=TRUE)
herbaceous_cover_rast <- load_product_rast(dinp, "HL_TSA_LNDLG_SMA_VSA-CAO")

herbaceous_cover_rast <- rast(fname)

herbaceous_cover_changes <- herbaceous_cover_rast$CHANGE
herbaceous_cover_year_of_change <- herbaceous_cover_rast["YEAR-OF-CHANGE"]
herbaceous_cover_changes <- get_change_band(herbaceous_cover_rast)
herbaceous_cover_year_of_change <- get_yoc_band(herbaceous_cover_rast)



# VALUE OF PEAK SEASON
#######################################################################

fname <- dir(dinp, ".*HL_TSA_LNDLG_SMA_VPS-CAO.vrt$", full.names=TRUE)

peak_rast <- rast(fname)
peak_rast <- load_product_rast(dinp, "HL_TSA_LNDLG_SMA_VPS-CAO")

peak_changes <- peak_rast$CHANGE
peak_year_of_change <- peak_rast["YEAR-OF-CHANGE"]
peak_changes <- get_change_band(peak_rast)
peak_year_of_change <- get_yoc_band(peak_rast)



Expand Down
7 changes: 4 additions & 3 deletions docs/output.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,9 +158,10 @@ The `.tif` files are only published when the `--save_tsa` parameter is set to `t
<summary>Output files</summary>

- `trend/`
- `mosaic/<PRODUCT>/`
- `<TILE>/`: .tif files that are part of the mosaic.
- `mosaic/`: Contains a single virtual raster file that combines the .tif files into the mosaic visualization.
- `mosaic/`
- `<PRODUCT>/`: Contains all files for a single product mosaic.
- `<PRODUCT>.vrt` Virtual raster file for the product. This can be opened in GIS systems to view the mosaic for the product.
- `<TILE>`: Contains the tile data used in `.vrt` file for this mosaic.
- `pyramid/<TREND_TYPE>/trend/<TILE>/`: Contains tile-wise pyramid visualizations for every trend analyzed in the workflow.

</details>
Expand Down
24 changes: 20 additions & 4 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -471,24 +471,40 @@ process {

For a comprehensive list of parameters see the FORCE documentation on [force-l2ps parameterization](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html#l2-param) and [force-higher-level (TSA) parameterization](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html).

#### Preprocessing (FORCE_PREPROCESS)

The following FORCE parameters can _not_ be set through `task.ext.args` in preprocessing (for the `FORCE_PREPROCESS` process):

- `FILE_QUEUE`: This pipeline processes one image per process, which does not require a queue for input files.
- `DIR_LEVEL2`, `DIR_LOG`, `DIR_PROVENANCE`, `DIR_TEMP`: FORCE directories must be defined within the process working directory.
- `FILE_DEM`: Set through process's input channels and derived from `params.dem`, see [digital elevation model](#digital-elevation-model-dem) for details.
- `USE_DEM_DATABASE`: Not implemented yet.
- `FILE_TILE`: Set through the process's input channels.
- `TILE_SIZE`: Derived from datacube input channel, which is created based on the [datacube input](#datacube).
- `BLOCK_SIZE`: Derived from datacube input channel, which is created based on the [datacube input](#datacube)..
- `ORIGIN_LON`: Derived from datacube input channel, which is created based on the [datacube input](#datacube)..
- `ORIGIN_LAT`: Derived from datacube input channel, which is created based on the [datacube input](#datacube)..
- `PROJECTION`: Derived from datacube input channel, which is created based on the [datacube input](#datacube)..
- `ORIGIN_LON`: Derived from datacube input channel, which is created based on the [datacube input](#datacube).
- `ORIGIN_LAT`: Derived from datacube input channel, which is created based on the [datacube input](#datacube).
- `PROJECTION`: Derived from datacube input channel, which is created based on the [datacube input](#datacube).
- `WVP`: Set through process's input channels and derived from `--wvdb`, see [water vapor database](#water-vapor-database-wvdb) for details.
- `DIR_AOD`: Set through process's input channels and derived from `--aod`, see [aerosol optical depth](#aerosol-optical-depth-aod) for details.
- `DIR_COREG_BASE`: Set through process's input channels and derived from `--coreg`, see [coregistration](#coregistration-near-infrared-nir-data) for details.
- `FILE_AOI`: Set through process's input channels and derived from `--aoi`, see [area of interest](#area-of-interest-aoi) for details.

The following parameter's default values deviate from their current FORCE default:

- `OUTPUT_FORMAT`: set to `GTiff`(GeoTIFF) instead of `COG` (cloud optimized GeoTIFF)

In addition, the `DO_TOPO` FORCE parameter will only be considered when the [pipeline is executed with a digital elevation model](#digital-elevation-model-dem).

> [!WARNING]
> Please refer to the FORCE documentation as mentioned above to understand the impact of different parameters.
> Certain combinations of parameters may break FORCE or the pipeline.
>
> The `OUTPUT_FORMAT` parameters and `FILE_OUTPUT_OPTIONS` should be modified with caution.
> The pipeline expects both modules to return `.tif` files. Other output formats will break the pipeline.
> In addition, certain combinations of these parameters may require custom containers (e.g. container with COG GDAL drivers for `OUTPUT_FORMAT` = `COG`.)

#### Higher-level processing (FORCE_HIGHER_LEVEL)

The following FORCE parameters can _not_ be set through `task.ext.args` in higher-level processing (for the `FORCE_HIGHER_LEVEL` process):

- `DIR_LOWER`, `DIR_HIGHER`, `DIR_PROVENANCE`: FORCE directories must be defined within the process working directory.
Expand Down
9 changes: 2 additions & 7 deletions modules/local/check_results/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,8 @@ process CHECK_RESULTS {

script:
"""
files=`find ./trend/ -maxdepth 1 -mindepth 1 -type d`
for path in \$files; do
mkdir -p trend/\$(ls \$path)
cp \$path/*/* trend/\$(ls \$path)/
rm \$path -r
done;
test.R trend/mosaic $woody_change_ref $woody_yoc_ref $herbaceous_change_ref $herbaceous_yoc_ref $peak_change_ref $peak_yoc_ref
# run check results
test.R ./trend $woody_change_ref $woody_yoc_ref $herbaceous_change_ref $herbaceous_yoc_ref $peak_change_ref $peak_yoc_ref

cat <<-END_VERSIONS > versions.yml
"${task.process}":
Expand Down
8 changes: 1 addition & 7 deletions modules/local/check_results_full/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,7 @@ process CHECK_RESULTS_FULL {

script:
"""
files=`find ./trend/ -maxdepth 1 -mindepth 1 -type d`
for path in \$files; do
mkdir -p trend/\$(ls \$path)
cp \$path/*/* trend/\$(ls \$path)/
rm \$path -r
done;
test.R trend/mosaic $reference
test.R ./trend $reference

cat <<-END_VERSIONS > versions.yml
"${task.process}":
Expand Down
2 changes: 1 addition & 1 deletion modules/local/force-generate_analysis_mask/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ process FORCE_GENERATE_ANALYSIS_MASK{
tag { aoi.simpleName }
label 'process_single'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
path aoi
Expand Down
2 changes: 1 addition & 1 deletion modules/local/force-generate_tile_allow_list/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ process FORCE_GENERATE_TILE_ALLOW_LIST{
tag { aoi.simpleName }
label 'process_single'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
path aoi
Expand Down
21 changes: 17 additions & 4 deletions modules/local/force-higher_level/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ process FORCE_HIGHER_LEVEL {
label 'process_medium'
label 'error_retry'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
tuple val(meta), path(boa), path(qai), path(mask)
path 'ard/datacube-definition.prj'
path cube
path endmember
path allow_list
val resolution
Expand Down Expand Up @@ -53,6 +53,7 @@ process FORCE_HIGHER_LEVEL {
def outputOptions = task.ext.args?.getAt("FILE_OUTPUT_OPTIONS") ? "FILE_OUTPUT_OPTIONS = ${task.ext.args["FILE_OUTPUT_OPTIONS"]}" : "FILE_OUTPUT_OPTIONS = NULL"
def outputExplode = task.ext.args?.getAt("OUTPUT_EXPLODE") ? "OUTPUT_EXPLODE = ${task.ext.args["OUTPUT_EXPLODE"]}" : "OUTPUT_EXPLODE = FALSE"
def outputSubDirs = "OUTPUT_SUBFOLDERS = FALSE"
def failIfEmpty = task.ext.args?.getAt("FAIL_IF_EMPTY") ? "FAIL_IF_EMPTY = ${task.ext.args["FAIL_IF_EMPTY"]}" : "FAIL_IF_EMPTY = FALSE"

// Parallel processing
def nThreadRead = task.ext.args?.getAt("NTHREAD_READ") ? "NTHREAD_READ = ${task.ext.args["NTHREAD_READ"]}" : "NTHREAD_READ = 8"
Expand All @@ -65,13 +66,14 @@ process FORCE_HIGHER_LEVEL {
def xTilePrm = "X_TILE_RANGE = $xTile $xTile"
def yTilePrm = "Y_TILE_RANGE = $yTile $yTile"
def fileTile = allow_list ? "FILE_TILE = $allow_list" : "FILE_TILE = NULL"
def blockSize = task.ext.args?.getAt("BLOCK_SIZE") ? "BLOCK_SIZE = ${task.ext.args["BLOCK_SIZE"]}" : "BLOCK_SIZE = 0"
def chunkSize = task.ext.args?.getAt("CHUNK_SIZE") ? "CHUNK_SIZE = ${task.ext.args["CHUNK_SIZE"]}" : "CHUNK_SIZE = 0 0"
def res = "RESOLUTION = $resolution"
def reducePSF = task.ext.args?.getAt("REDUCE_PSF") ? "REDUCE_PSF = ${task.ext.args["REDUCE_PSF"]}" : "REDUCE_PSF = FALSE"
def useL2Improph = task.ext.args?.getAt("USE_L2_IMPROPHE") ? "USE_L2_IMPROPHE = ${task.ext.args["USE_L2_IMPROPHE"]}" : "USE_L2_IMPROPHE = FALSE"

// Sensor allow list
def sensors = "SENSORS $sensors_level2"
def targetSensor = task.ext.args?.getAt("TARGET_SENSOR") ? "TARGET_SENSOR = ${task.ext.args["TARGET_SENSOR"]}" : "TARGET_SENSOR = LNDLG"
def productTypeMain = task.ext.args?.getAt("PRODUCT_TYPE_MAIN") ? "PRODUCT_TYPE_MAIN = ${task.ext.args["PRODUCT_TYPE_MAIN"]}" : "PRODUCT_TYPE_MAIN = BOA"
def productTypeQuality = task.ext.args?.getAt("PRODUCT_TYPE_QUALITY") ? "PRODUCT_TYPE_QUALITY = ${task.ext.args["PRODUCT_TYPE_QUALITY"]}" : "PRODUCT_TYPE_QUALITY = QAI"
def spectralAdjust = task.ext.args?.getAt("SPECTRAL_ADJUST") ? "SPECTRAL_ADJUST = ${task.ext.args["SPECTRAL_ADJUST"]}" : "SPECTRAL_ADJUST = FALSE"
Expand Down Expand Up @@ -170,7 +172,16 @@ process FORCE_HIGHER_LEVEL {
mkdir -p $ardPath
mv *.tif $ardPath


# set chunk size
CHUNKSIZE='${chunkSize}'
if [[ "\$CHUNKSIZE" == "CHUNK_SIZE = 0 0" ]]; then
TILESIZE=\$(sed '6q;d' $cube)
CHUNKSIZE="CHUNK_SIZE = \$TILESIZE"
fi

# create parameter file
mv "$cube" "${ardBasePath}datacube-definition.prj"

PARAM=./tsa_${meta.id}.prm
cat <<EOF > \$PARAM
Expand All @@ -184,6 +195,7 @@ process FORCE_HIGHER_LEVEL {
${outputOptions}
${outputExplode}
${outputSubDirs}
${failIfEmpty}
${nThreadRead}
${nThreadCompute}
${nThreadWrite}
Expand All @@ -192,11 +204,12 @@ process FORCE_HIGHER_LEVEL {
${xTilePrm}
${yTilePrm}
${fileTile}
${blockSize}
\$CHUNKSIZE
${res}
${reducePSF}
${useL2Improph}
${sensors}
${targetSensor}
${productTypeMain}
${productTypeQuality}
${spectralAdjust}
Expand Down
2 changes: 1 addition & 1 deletion modules/local/force-mosaic/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ process FORCE_MOSAIC{
tag { meta.id }
label 'process_low'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
tuple val(meta), path('trend/*')
Expand Down
9 changes: 5 additions & 4 deletions modules/local/force-preprocess/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ process FORCE_PREPROCESS {
label 'process_medium'
label 'error_retry'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
tuple val(meta), path(data)
Expand Down Expand Up @@ -53,14 +53,15 @@ process FORCE_PREPROCESS {

// Digital elevation model
// "FILE_DEM" can only be set after input stage-in
def useDemDB = "USE_DEM_DATABASE = FALSE"
def demResampling = task.ext.args?.getAt("DEM_RESAMPLING") ? "DEM_RESAMPLING = ${task.ext.args["DEM_RESAMPLING"]}" : "DEM_RESAMPLING = BL"
def demNoData = task.ext.args?.getAt("DEM_NODATA") ? "DEM_NODATA = ${task.ext.args["DEM_NODATA"]}" : "DEM_NODATA = -32767"

// Data cubes
def doReproj = task.ext.args?.getAt("DO_REPROJ") ? "DO_REPROJ = ${task.ext.args["DO_REPROJ"]}" : "DO_REPROJ = TRUE"
def doTile = task.ext.args?.getAt("DO_TILE") ? "DO_TILE = ${task.ext.args["DO_TILE"]}" : "DO_TILE = TRUE"
def fileTile = tile ? "FILE_TILE = $tile" : "FILE_TILE = NULL"
// "TILE_SIZE" can only be set after input stage-in
// "BLOCK_SIZE" can only be set after input stage-in
def landsatRes = task.ext.args?.getAt("RESOLUTION_LANDSAT") ? "RESOLUTION_LANDSAT = ${task.ext.args["RESOLUTION_LANDSAT"]}" : "RESOLUTION_LANDSAT = 30"
def sentinel2Res = task.ext.args?.getAt("RESOLUTION_SENTINEL2") ? "RESOLUTION_SENTINEL2 = ${task.ext.args["RESOLUTION_SENTINEL2"]}" : "RESOLUTION_SENTINEL2 = 10"
// "ORIGIN_LON can only be set after input stage-in
Expand Down Expand Up @@ -150,7 +151,6 @@ process FORCE_PREPROCESS {
ORIGINX=\$(sed '2q;d' $cube)
ORIGINY=\$(sed '3q;d' $cube)
TILESIZE=\$(sed '6q;d' $cube)
BLOCKSIZE=\$(sed '7q;d' $cube)

# create parameter file
PARAM=./preprocess_${data.simpleName}.prm
Expand All @@ -165,11 +165,12 @@ process FORCE_PREPROCESS {
${fileAoi}
FILE_DEM = \$DEM_FILE
${demNoData}
${useDemDB}
${demResampling}
${doReproj}
${doTile}
${fileTile}
TILE_SIZE = \$TILESIZE
BLOCK_SIZE = \$BLOCKSIZE
${landsatRes}
${sentinel2Res}
ORIGIN_LON = \$ORIGINX
Expand Down
2 changes: 1 addition & 1 deletion modules/local/force-pyramid/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ process FORCE_PYRAMID {
tag { meta.id }
label 'process_low'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
tuple val(meta), path(image)
Expand Down
2 changes: 1 addition & 1 deletion modules/local/merge/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ process MERGE {
label 'process_low'
label 'error_retry'

container "nf-core/force:3.8.01"
container "nf-core/force:3.10.04"

input:
val (data_type) // defines whether qai or boa is merged
Expand Down
2 changes: 1 addition & 1 deletion nf-test.config
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ config {

// load the necessary plugins
plugins {
load "nft-utils@0.0.3"
load "nft-utils@0.0.9"
}
}
Loading
Loading