Updated from master, and added hsofs grid

Author: jmpmcmanus

README.md

@@ -1,15 +1,16 @@
 <img src="adds/imgs/kalpana.PNG" width="1028"/>
 
-Kalpana is a Python module to convert *ADCIRC* output files to geospatial vector formats (shapefile or kmz) and to downscale the maximum water elevation *maxele.6.nc*. 
+Kalpana is a Python module to convert *ADCIRC* output files to geospatial vector formats (shapefile or kmz), and to downscale the maximum water elevations onto a higher-resolution raster. 
 
-## *ADCIRC* to geospatial vector files
+## *ADCIRC* to geospatial vector formats
 
-Kalpana is capable of converting time-varying outputs such as the significant wave height and time-constant outputs such as the maximum water elevation into polylines or polygons in both of the recently mentioned formats.
-Kalpana was originally built by Rosemary Cyriac, and her efforts were aided by the initial attempts of Rich Signell (USGS) and Rusty Holleman to generate shapefiles from ADCIRC results. Then, Jason Fleming improved Kalpana and incorporated the code into ADCIRC Surge Guidance System (ASGS) that he maintains at the Renaissance Computing Institute (RENCI).
+Kalpana can convert time-varying outputs (e.g. *fort.63.nc*, *swan_HS.63.nc*) and time-constant outputs (*maxele.63.nc*) into polylines or polygons in both shapefile and kmz formats. 
+Kalpana was developed originally by Rosemary Cyriac, and her efforts were aided by the work of Rich Signell and Rusty Holleman to generate shapefiles from *ADCIRC* results. Then, Jason Fleming improved Kalpana and incorporated it into the ADCIRC Surge Guidance System (ASGS).
 
 ## Downscaling
 
-Kalpana transforms the *ADICRC* *maxele.63.nc* file to a constant and higher resolution DEM considering small scale topographic/bathymetric features. The static downscaled method was developed by Nelson Tull, and the incorporated into Kalpana along with a new downscaling method by Carter Rucker. The details can be found in [this paper](https://link.springer.com/epdf/10.1007/s11069-021-04634-8?sharing_token=5GBxenc0qDVGHm3BGk6KhPe4RwlQNchNByi7wbcMAY69maaLpgXTBxca-OorPGWBn2w2ySSkXhIRhNeWoyNx8-ituX0UqAcNj_LDMh_kFz6sCpb5e882TbeHKiKpzRd_j4XfVH_6ONriheKYxx2CECQI07z23OD-pFrCALWfyVc=). The schematics below show the downscaling process.
+Kalpana can downscale the maximum water elevations (*maxele.63.nc*) to a higher-resolution raster by considering small-scale topographic features. This process can provide a more-accurate representation of the inundation extent. 
+As part of the downscaling, the water surface can be expanded outward to intersect with the ground surface, beyond the extent predicted by ADCIRC. This expansion can be done in two ways: the static method was developed by Nelson Tull, and then the head-loss method was developed by Carter Rucker. The details can be found in [this paper](https://link.springer.com/epdf/10.1007/s11069-021-04634-8?sharing_token=5GBxenc0qDVGHm3BGk6KhPe4RwlQNchNByi7wbcMAY69maaLpgXTBxca-OorPGWBn2w2ySSkXhIRhNeWoyNx8-ituX0UqAcNj_LDMh_kFz6sCpb5e882TbeHKiKpzRd_j4XfVH_6ONriheKYxx2CECQI07z23OD-pFrCALWfyVc=). The schematics below show the downscaling process.
 
 **Storm surge expansion**
 
@@ -21,6 +22,6 @@ Kalpana transforms the *ADICRC* *maxele.63.nc* file to a constant and higher res
 
 ## Updated version
 
-Kalpana was updated to python 3 and upgraded by Tomás Cuevas as a part of his MSc research.
-Instructions for using Kalpana can be found in the examples folder.
-For any question, comment or suggestion please send an email to tomascuevas@gmail.com or open an *Issue*
+Kalpana was updated to python 3 and upgraded by Tomás Cuevas as a part of his MSc research. 
+Instructions for using Kalpana can be found in the examples folder, including a few Jupyter notebooks created by Brandon Tucker. 
+For any question, comment or suggestion please send an email to tomascuevas@gmail.com or open an *Issue*.

examples/README.md

@@ -2,8 +2,8 @@
 
 *Kalpana* has two main capabilities:
 
-1. Visualization: export *ADCIRC* output as geospatial vector data for visualization using QGIS or any similar GIS software.
-2. Downscaling: transform the *ADICRC* *maxele.63.nc* file to a constant and higher resolution DEM considering small scale topographic/bathymetric features (Rucker et. al 2021).
+1. Visualization: convert *ADCIRC* output as geospatial vector data for visualization using QGIS or any similar GIS software.
+2. Downscaling: transform the *ADCIRC* *maxele.63.nc* file to a higher-resolution raster by considering small-scale topographic features (Rucker et. al 2021).
 
 ## Visualization 
 
@@ -12,37 +12,37 @@ Setup for running the visualization examples (export_exampleXX.ipynb):
 2. Clone the Kalpana repository to your local device. [Here](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) you can find the instructions.
 
 **Export example 01** (export_example01.ipynb)<br>
-Create contours as polygons based on the maximum flooding outputs from ADCIRC, then export the polygons as a .shp file.
+Create contours as polygons based on the maximum flooding outputs from *ADCIRC*, then export the polygons as a .shp file.
 
 **Export example 02** (export_example02.ipynb)<br>
-Create contours as polylines based on the maximum flooding outputs from ADCIRC, then export the polylines as a .shp file.
+Create contours as polylines based on the maximum flooding outputs from *ADCIRC*, then export the polylines as a .shp file.
 
 **Export example 03** (export_example03.ipynb)<br>
-Read fort.14 file as a GeoPandas GeoDataFrame, then export it as .shp file and some visualizations.
+Read *fort.14* file as a GeoPandas GeoDataFrame, then export it as .shp file and some visualizations.
 
 **More examples are comming soon**
 
 ## Downscaling
 
-To use the downscaling method, it is required to have *GRASS GIS* installed. *Kalpana* uses *GRASS GIS* in the backend calling it from a python script. On a Linux OS, you just need to install/compile *GRASS GIS* (https://grass.osgeo.org/) and create a [conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) using the yml file provided in the *install* folder. On Windows it is more complicated. You need to install *GRASS GIS* and use the *Python* installation that comes with *GRASS GIS*. We will provide a short tutorial soon. To make it more user-friendly we made two *Docker* images to run the *Kalpana* downscaling tools. One of the images if for running the container interactively, and the other is non interactive. The instructions for using them are listed below:
+To use the downscaling method, it is required to have *GRASS GIS* installed. *Kalpana* uses *GRASS GIS* in the backend calling it from a python script. On a Linux OS, you just need to install/compile *GRASS GIS* (https://grass.osgeo.org/) and create a [conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) using the yml file provided in the *install* folder. On Windows it is more complicated. You need to install *GRASS GIS* and use the *Python* installation that comes with *GRASS GIS*. We will provide a short tutorial soon. 
+
+To make it more user-friendly we made two *Docker* images to run the *Kalpana* downscaling tools. With these images, a user can skip the software installation and proceed directly to using *Kalpana*. One image is for running the container interactively, and the other image is non-interactive. The instructions for using them are listed below:
 
 **Non interactive**<br>
-This image has all the necessary files and has been set up to downscale *ADCIRC* simulations using *NC9* mesh on a DEM of North Carolina. It is configured to run automatically, i.e. when running the container, the downscaling scripts are executed.
+This image has all the necessary files and has been set up to downscale *ADCIRC* simulations using the *NC9* mesh on a DEM of North Carolina. It is configured to run automatically, i.e. when running the container, the downscaling scripts are executed.
 1. Install Docker, follow instructions [here](https://docs.docker.com/engine/install/).
-2. Using the terminal, pull the Docker image from Docker hub with the command below. This image can be used only for running the downscaling for a simulation donw with NC9 in North Carolina. <br>
+2. Using the terminal, pull the Docker image from Docker hub with the command below. This image can be used only for running the downscaling for a simulation done with the NC9 mesh in North Carolina. <br>
     ```docker pull tacuevas/kalpana_nc:latest```
 3. Create a folder, place the maxele.63.nc and runKalpanaStatic.inp files inside, and 'cd' to it. The *inp* file is provided in this folder, and the *ADCIRC* *maxele.63.nc* file can be found [here](https://go.ncsu.edu/kalpana-example-inputs).
 4. Modify the file *runKalpanaStatic.inp* if you want to change the downscaling inputs (e.g. levels, crs, vertical unit, etc).
-5. Run the container declaring a volume so kalpana can access the folder created in *step 3*. Before running the container, check you are located in the same folder where you placed the input files. We also provide a copy of the *Python* script executed with the container is ran (*runKalpanaStatic.py*)<br>
+5. Run the container declaring a volume so kalpana can access the folder created in *step 3*. Before running the container, check you are located in the same folder where you placed the input files. We also provide a copy of the *Python* script executed when the container is ran (*runKalpanaStatic.py*)<br>
     ```docker run -it -v "$(pwd)":/home/kalpana/inputs tacuevas/kalpana_nc:latest```
 
-
 We are working to implement this same image for other areas of the US.
 
 **Interactive**<br>
 This image is configured to run kalpana interactively, all the python packages and *GRASS GIS* are installed. You need to copy the examples *downscaling_exampleXX.py* , the necessary inputs (availables [here](https://drive.google.com/drive/folders/1cbQzN4SrLs_rVlz9q8zHCKbFtQpLO5CG?usp=sharing)), and the *Kalpana* *downscaling.py* and *export.py* python modules from this repo to the container.
 
-
 The steps for running the container:
 
 1) Install Docker, follow instructions [here](https://docs.docker.com/engine/install/).
@@ -57,12 +57,12 @@ The steps for running the container:
 Each example is explained below, remember to modify the paths!
 
 **Example 01** (downscaling_example01.py)<br>
-This script creates a grass location importing the DEM for downscaling and also creates a new DEM with same resolution and extend with the size of the mesh triangles. This step is key for the downscaling and can be run in advance, since it only depends on the mesh (*fort.14*).
+This script creates a grass location by importing the DEM for downscaling. It also creates a new DEM with the same resolution and extent, but instead of the ground surface, the DEM represents the sizes of the mesh elements. This step is key for the downscaling and can be run in advance, because it only depends on the mesh (*fort.14*).
 
 **Example 02** (downscaling_example02.py)<br>
-Example for doing the static downscaling using an existing grass location, and importing the DEM with the mesh elements size. Both inputs were created in the *example 01*. There is a short description of all inputs in the script, more detail can be found in the docstring of the function in the github repository.
+Example for doing the static downscaling using an existing grass location, and importing the DEM with the mesh element sizes. Both inputs were created in the *example 01*. There is a short description of all inputs in the script, more detail can be found in the docstring of the function in the github repository.
 
 **Example 03** (downscaling_example03.py)<br>
-This example combines downscaling examples *1* and *2*. The DEM with mesh elements size and the grass location are created. This should be considerable more slow than running *example 2*, since creating the inputs for the downscaling is the slower part.
+This example combines downscaling examples *1* and *2*. The DEM with mesh elements size and the grass location are created. This should be considerably more slow than running *example 2*, because creating the inputs for the downscaling is the slower part.
 
-For questions regarding the visualization or downscaling examples please open an *Issue* or email tomascuevas@gmail.com
+For questions regarding the visualization or downscaling examples please open an *Issue* or email tomascuevas@gmail.com.

kalpana/downscaling.py

@@ -22,12 +22,6 @@
     EXPLAIN WORKFLOW
 '''
 
-def call_proc(cmd):
-    # This runs in a separate thread
-    p = Popen(cmd, stdout=PIPE, stderr=STDOUT)
-    stdout, stderr = p.communicate()
-    return (stdout, stderr)
-
 def delFiles(listFiles, typeFiles, pkg):
     ''' Delete files form a grass location
         Parameters
@@ -733,14 +727,15 @@ def runStatic(ncFile, levels, epsgOut, pathOut,  grassVer, pathRasFiles, rasterF
 
     if rasterFiles == 'all':
         rasterFiles = os.listdir(pathRasFiles)
+
     ## setup grass env
     setGrassEnv(grassVer, pathGrassLocation, createGrassLocation, gs, gsetup,
                 pathRasFiles, rasterFiles, createLocMethod, epsgOut)
 
     t2 = time.time()
     logger.info(f'    Setup grass environment: {(t2 - t11)/60:0.2f} min') # Changed
 
-     ## setup growing
+    ## setup growing
     logger.info(f'    Start Downscaling preprocess') # Changed
     ## here the thres must be in square meters
     setupGrowing(pathOut, attrCol, exportMesh, meshFile, thres, gs, epsgOut, exportOrg)
@@ -781,7 +776,7 @@ def meshRepLen2raster(fort14, epsgIn, epsgOut, pathOut, grassVer, pathRasFiles,
             pathRasters: str
                 path of the raster files
             rasterFiles: list or str
-                name(s) of the raster file(s).
+                name(s) of the raster file(s). If 'all' is input, all files in pathRasters are used
             subDomain: str or list. Default None
                 complete path of the subdomain polygon kml or shapelfile, or list with the
                 uper-left x, upper-left y, lower-right x and lower-right y coordinates. The crs must be the same of the
@@ -827,6 +822,10 @@ def meshRepLen2raster(fort14, epsgIn, epsgOut, pathOut, grassVer, pathRasFiles,
 
     logger.info(f'    Start Setup grass environment') # Changed
     t11 = time.time()
+
+    if rasterFiles == 'all':
+        rasterFiles = os.listdir(pathRasFiles)
+
     ## setup grass env
     setGrassEnv(grassVer, pathGrassLocation, createGrassLocation, gs, gsetup,
                 pathRasFiles, rasterFiles, createLocMethod, epsgOut)
@@ -874,22 +873,7 @@ def reprojectRas(filein, pathout, epsgOut=None, res='same'):
     if res == 'same':
         rasOut = rasIn.rio.reproject(epsgOut)
         logger.info('Reproject '+bname+' to EPSG 4326') # Changed
-        rasOut.rio.to_raster(os.path.join(pathout, bname + f'_epsg{epsgOut}.tmp.tif')) # Changed
-        program_list = [] # Changed
-        logger.info('Create program_list to run gdal_translate, and convert TIFF to COG')
-        program_list.append(['gdal_translate',os.path.join(pathout, bname + f'_epsg{epsgOut}.tmp.tif'),os.path.join(pathout, bname + f'_epsg{epsgOut}.tif'),'-of','COG','-co','COMPRESS=LZW']) # Changed
-         
-        # Apply cmds_list to pool, and output to resutls
-        for program in program_list: 
-            logger.info('Run '+" ".join(program))
-            logger.info('Convert '+os.path.join(pathout, bname + f'_epsg{epsgOut}.tmp.tif')+' to COG with filename '+os.path.join(pathout, bname + f'_epsg{epsgOut}.tif'))
-            try:
-                output = subprocess.run(program, shell=False, check=True)
-            except Exception as e:
-                logger.info(e)
-                sys.exit(0)
-            logger.info('Ran '+" ".join(program)+" with output returncode "+str(output.returncode))
-
+        rasOut.rio.to_raster(os.path.join(pathout, bname + f'_epsg{epsgOut}.tif'), driver="COG") # Changed
         logger.info('Wrote '+bname+' to COG') # Changed
     ## change resolution
     else: