diff --git a/.gitignore b/.gitignore index 57828a2..bea3b29 100644 --- a/.gitignore +++ b/.gitignore @@ -27,12 +27,12 @@ __pycache__/ # Packaging / build artifacts build/ -dist/ downloads/ .eggs/ eggs/ develop-eggs/ sdist/ +dist/ *.egg-info/ .installed.cfg *.egg @@ -41,6 +41,7 @@ pip-log.txt pip-delete-this-directory.txt # Test / coverage / tooling +test-results/ .tox/ .nox/ .pytest_cache/ diff --git a/README.md b/README.md deleted file mode 100644 index e441c44..0000000 --- a/README.md +++ /dev/null @@ -1,195 +0,0 @@ -# AEM Interpretation Conversion Tool (AEMInterpConvert) - -## Overview -For more information about AEMInterpConvert, Geoscience Australia’s Online Airborne Electromagnetic Interpretation Conversion Tool, see this [Geoscience Australia article](https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/150529). - ---- - -## Installation Guide - -### 1. Clone the Repository -```bash -git clone git@bitbucket.org:geoscienceaustralia/aem-interpretation-conversion.git -cd aem-interpretation-conversion -``` - ---- - -### 2. Choose Your Environment Setup - -### **Option A: Anaconda (Recommended)** -If you have [Anaconda](https://www.anaconda.com/download) installed: -```bash -conda env create -f environment.yml -conda activate aemworkflow-env -``` - ---- - -### **Option B: Python 3.12 + Virtual Environment (VENV)** -#### i. Ensure Python 3.12 is installed. - -**Linux / macOS:** -```bash -python3.12 -m venv aem-venv -source aem-venv/bin/activate -``` - -**Windows (Command Prompt):** -```powershell -python -m venv aem-venv -aem-venv\Scripts\activate.bat -``` - -#### ii. Install GDAL Dependencies. - -**Linux / macOS:** -Use the provided build script to install system-level GDAL and PROJ libraries: -```bash -cd builds -./linux_gdal.sh -``` - -This script will: -- Install required system packages (`gdal`, `proj`) -- Install the Python GDAL bindings matching your GDAL version - ---- - -**Windows** -1. Download and install [OSGeo4W](https://trac.osgeo.org/osgeo4w/). - - Choose **Advanced Install** - - Select the following packages: - - `gdal` - - `proj` - -2. After installation (default path is `C:\OSGeo4W` or `C:\OSGeo4W64`), run the helper script to configure environment variables and install Python GDAL bindings using Windows Command prompt (CMD): -```cmd -cd builds -windows_gdal.bat -``` - -This script will: -- Add OSGeo4W binaries to your `PATH` -- Detect the installed GDAL version (`gdalinfo --version`) -- Install the matching Python GDAL bindings via `pip` - ---- -#### iii. Verify GDAL Installation -To confirm GDAL bindings are working correctly: -```bash -python -c "from osgeo import gdal; print(gdal.VersionInfo())" -``` - -#### iv. Install the AEMInterpConvert Package - -After setting up dependencies: -```bash -pip install . -``` - - -### 3. CLI Usage ------------- -For each script, run the file with any required arguments and any additional where you want to deviate from the default. All arguments should be in quotes as per the examples. - -If using Anaconda, activate conda environment if required before running the scripts. - -### Pre-interpretation - -``` -aemworkflow pre-interpret "{input_directory}" "{output_directory}" -``` - -**Parameter examples:** - -| Flag | Argument | Required | Default |Options |Notes | -| ------------|-----------------| ------------ |-----------|----------|--------| -| | input directory | Yes |None | |A non zipped folder containing required files| -| | output directory| Yes |None | | | -| | crs - GDA/MGA zone EPSG| No |28349 |28349, 28350, 28351, 28352, 28354, 28355, 28356| | -| | Interpretation GIS software| No|esri_arcmap_0.5|esri_arcmap_0.5 or esri_arcmap_pro_0.5| | -| | depth lines | No |10 | | | -| | depth lines increment| No |30 | | | - -### Interpretation - -``` -aemworkflow interpret "{input_directory}" "{output_directory}" -``` -**Parameter examples:** - -| Flag | Argument | Required | Default |Options |Notes | -| ------------|-----------------| ------------ |-----------|----------|--------| -| | input directory | Yes |None | |A non zipped folder containing required files| -| | output directory| Yes |None | | | -| | crs - GDA/MGA zone EPSG| No |28349 |28349, 28350, 28351, 28352, 28354, 28355, 28356| | -| | Interpretation GIS software| No|esri_arcmap_0.5|esri_arcmap_0.5 or esri_arcmap_pro_0.5| | -| | depth lines | No |10 | | | -| | depth lines increment| No |30 | | | - -### Validation - -``` -aemworkflow validate "{input_directory}" "{output_directory}" "{asud_filename}" -``` -**Parameter examples:** - -| Flag | Argument | Required | Default |Options |Notes | -| ------------|-----------------| ------------ |-----------|----------|--------| -| | input directory | Yes |None | |A non zipped folder containing required files| -| | output directory| Yes |None | | | -| | ausd file name | Yes |None | | | - - -### Conversion - -``` -aemworkflow convert "{input_directory}" "{output_directory}" -``` -**Parameter examples:** - -| Flag | Argument | Required | Default |Options |Notes | -| ------------|-----------------| ------------ |-----------|----------|--------| -| | input directory | Yes |None | |A non zipped folder containing required files| -| | output directory| Yes |None | | | -| | crs - GDA/MGA zone EPSG| No |28349 |28349, 28350, 28351, 28352, 28354, 28355, 28356| | - - -### Export - -``` -aemworkflow export "{input_directory}" "{output_directory}" "{boundary_file}" "{split_file}" -mdc -mdch -egs -``` -**Parameter examples:** - -| Flag | Argument | Required | Default |Options |Notes | -| ------------|-----------------| ------------ |-----------|----------|--------| -| | input directory | Yes |None | |A non zipped folder containing required files| -| | output directory| Yes |None | | | -| | name of boundary file| Yes |None | | | -| | name of split file | Yes |None | | | -| -mdc | Export to MDC format | Yes |False | | | -| -mdch | Export to MDCH format| Yes |False | | | -| -egs | Export to EGS format | Yes |False | | | - - -Useful Links ------------- - -Home Page - http://github.com/GeoscienceAustralia/aem-interpretation-conversion - -Documentation - http://GeoscienceAustralia.github.io/aem-interpretation-conversion - -Issue tracking - https://github.com/GeoscienceAustralia/aem-interpretation-conversion/issues - -AEMInterpConvert website user interface https://aem.toolkits.ga.gov.au - -Bugs & Feedback ---------------- - -For bugs, questions and discussions, please use -Github Issues diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..b8b2ec7 --- /dev/null +++ b/README.rst @@ -0,0 +1,264 @@ +AEM Interpretation Conversion Tool (AEMInterpConvert) +================================================================ + +Background +========== + +This online Airborne Electromagnetic (AEM) data viewer is designed to provide a user-friendly interface for visualizing and analyzing AEM data. + +For more information please see the `Metadata Statement and User Guide`_ and `Workflow Document`_. + +.. _Metadata Statement and User Guide: https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/150529 + +.. _Workflow Document: https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/147251 + + +Installation +============ + +1. Clone the Repository +------------------------------- +.. code-block:: bash + + git clone git@bitbucket.org:geoscienceaustralia/aem-interpretation-conversion.git + cd aem-interpretation-conversion + +--- + +2. Choose Your Environment Setup +-------------------------------------- + + +**Option A: Anaconda (Recommended)** +---------------------------------------------- + +If you have `Anaconda`_ installed + +.. _Anaconda: https://www.anaconda.com/downloads + +.. code-block:: bash + + conda env create -f environment.yml + conda activate aemworkflow-env + + +**Option B: Python 3.12 + Virtual Environment (VENV)** +---------------------------------------------- +Install a virtual environment +-------------------------------- + +Ensure Python 3.12 is installed on your system. +You can download it from the official Python website: https://www.python.org/downloads/. + +.. _Linux/MacOS instructions: + +Linux/MacOS instructions +---------------- +To create a virtual environment, open a terminal and run the following commands: + +.. code-block:: bash + + python3.12 -m venv aem-venv + source aem-venv/bin/activate + +This will create a virtual environment named `aem-venv` and activate it. + +Then install the GDAL dependencies: +Use the provided build script to install system-level GDAL and PROJ libraries: + +.. code-block:: bash + + ./builds/linux_gdal.sh + +This script will: + +- Install required system packages (gdal, proj) +- Install the Python GDAL bindings matching your GDAL version + +And add the current directory to the Python path: + +.. code-block:: bash + + export PYTHONPATH=$PYTHONPATH:$(pwd) + +Continue with the verification and installation instructions below + +.. _Windows instructions: + +Windows instructions +---------------- +To create a virtual environment, open a terminal and run the following commands: + +.. code-block:: bash + python -m venv aem-venv + aem-venv\Scripts\activate.bat + +Download and install `OSGeo4W`_. + +.. _OSGeo4W: https://trac.osgeo.org/osgeo4w/ + +- Choose Advanced Install +- Select the following packages: + - gdal + - proj +After installation (default path is C:\OSGeo4W or C:\OSGeo4W64), run the helper script to configure environment variables and install Python GDAL bindings using Windows Command prompt (CMD): + +.. code-block:: bash + + cd builds + windows_gdal.bat + +This script will: + +- Add OSGeo4W binaries to your PATH +- Detect the installed GDAL version (gdalinfo --version) +- Install the matching Python GDAL bindings via pip + +Continue with the verification and installation instructions below + +.. _verification-and-installation: + +Verification and installation +---------------- + +Then verify the gdal installation and bindings are working correctly by ensuring this command runs without errors: + +.. code-block:: bash + python -c "from osgeo import gdal; print(gdal.VersionInfo())" + +Finally, install the AEMInterpConvert Package: + +.. code-block:: bash + + pip install . + + +Run scripts +============ + +For each script, run the file with any required arguments and any additional where you want to deviate from the default. All arguments should be in quotes as per the examples. + +If using Anaconda, activate conda environment if required before running the scripts. + +Pre-interpretation +----------------------- + + +.. code-block:: bash + + aemworkflow pre-interpret --i "{input_directory}" --o "{output_directory}" + + +**Parameter examples:** + + +============================= ============== =============== ================================================ ============================================= +Argument Required Default Options Notes +============================= ============== =============== ================================================ ============================================= +input directory Yes None A non zipped folder containing required files +output directory Yes None +coordinate reference system No 28349 28349, 28350, 28351, 28352, 28354, 28355, 28356 GDA/MGA zone EPSG +GIS software No Esri ArcMap esri_arcmap_0.5 or esri_arcmap_pro_0.5 +number of depth lines No 10 +lines increments in metres No 30 +============================= ============== =============== ================================================ ============================================= + +Interpretation +----------------------- + + +.. code-block:: bash + + aemworkflow interpret --i "{input_directory}" --o "{output_directory}" + +**Parameter examples:** + +============================= ============== =============== ================================================ ============================================= +Argument Required Default Options Notes +============================= ============== =============== ================================================ ============================================= +input directory Yes None A non zipped folder containing required files +output directory Yes None +coordinate reference system No 28349 28349, 28350, 28351, 28352, 28354, 28355, 28356 GDA/MGA zone EPSG +GIS software No Esri ArcMap esri_arcmap_0.5 or esri_arcmap_pro_0.5 +number of depth lines No 10 +lines increments in metres No 30 +============================= ============== =============== ================================================ ============================================= + +Validation +----------------------- + +.. code-block:: bash + + aemworkflow validate --i "{input_directory}" --o "{output_directory}" --a "{asud_filename}" + +**Parameter examples:** + +============================= ============== =============== ========= ============================================= +Argument Required Default Options Notes +============================= ============== =============== ========= ============================================= +input directory Yes None A non zipped folder containing required files +output directory Yes None +asud filename Yes None +============================= ============== =============== ========= ============================================= + +Conversion +----------------------- + +.. code-block:: bash + + aemworkflow convert --i "{input_directory}" --o "{output_directory}" + +**Parameter examples:** + +============================= ============== =============== ================================================ ============================================= +Argument Required Default Options Notes +============================= ============== =============== ================================================ ============================================= +input directory Yes None A non zipped folder containing required files +output directory Yes None +coordinate reference system No 28349 28349, 28350, 28351, 28352, 28354, 28355, 28356 GDA/MGA zone EPSGac +============================= ============== =============== ================================================ ============================================= + + +Export +----------------------- + +.. code-block:: bash + + aemworkflow export --i "{input_directory}" --o "{output_directory}" --b "{boundary_file}" --s "{split_file}" --mdc --mdch --egs + +**Parameter examples:** + +============================= ============== =============== ================================================ ============================================= +Argument Required Default Options Notes +============================= ============== =============== ================================================ ============================================= +input directory Yes None A non zipped folder containing required files +output directory Yes None +asud filename Yes None +boundary file Yes None +split file No None +--mdc No False Add the flag if you want to set to true Export to MDC format +--mdch No False Add the flag if you want to set to true Export to MDCH format +--egs No False Add the flag if you want to set to true Export to EGS format +============================= ============== =============== ================================================ ============================================= + + + +Useful Links +------------ + +Home Page + http://github.com/GeoscienceAustralia/aem-interpretation-conversion + +Documentation + http://GeoscienceAustralia.github.io/aem-interpretation-conversion + +Issue tracking + https://github.com/GeoscienceAustralia/aem-interpretation-conversion/issues + +AEMInterpConvert website user interface https://aem.toolkits.ga.gov.au + +Bugs & Feedback +--------------- + +For bugs, questions and discussions, please use +Github Issues https://github.com/GeoscienceAustralia/aem-interpretation-conversion/issues diff --git a/aemworkflow/aemworkflow.py b/aemworkflow/aemworkflow.py index a099a73..16deff2 100644 --- a/aemworkflow/aemworkflow.py +++ b/aemworkflow/aemworkflow.py @@ -12,8 +12,8 @@ def cli(): pass @cli.command() -@click.argument("input_directory", type=click.Path(exists=True)) -@click.argument("output_directory", type=click.Path()) +@click.option("--i", "input_directory", type=click.Path(exists=True), required=True) +@click.option("--o", "output_directory", type=click.Path(), required=True) @click.option("--crs", default="28349", help="Coordinate Reference System (default: EPSG:28349)") @click.option("--gis", default="esri_arcmap_0.5", help="GIS format (default: esri_arcmap_0.5)") @click.option("--lines", default=10, help="Number of depth lines (default: 10)") @@ -27,8 +27,8 @@ def pre_interpret(input_directory, output_directory, crs, gis="esri_arcmap_0.5", sys.exit(1) @cli.command() -@click.argument("input_directory", type=click.Path(exists=True)) -@click.argument("output_directory", type=click.Path()) +@click.option("--i", "input_directory", type=click.Path(exists=True), required=True) +@click.option("--o", "output_directory", type=click.Path(), required=True) @click.option("--crs", default="28349", help="Coordinate Reference System (default: EPSG:28349)") @click.option("--gis", default="esri_arcmap_0.5", help="GIS format (default: esri_arcmap_0.5)") @click.option("--lines", default=10, help="Number of depth lines (default: 10)") @@ -42,9 +42,9 @@ def interpret(input_directory, output_directory, crs="28349", gis="esri_arcmap_0 sys.exit(1) @cli.command() -@click.argument("input_directory", type=click.Path(exists=True)) -@click.argument("output_directory", type=click.Path()) -@click.argument("asud_filename", type=click.Path(exists=True)) +@click.option("--i", "input_directory", type=click.Path(exists=True), required=True) +@click.option("--o", "output_directory", type=click.Path(), required=True) +@click.option("--a", "asud_filename", type=click.Path(exists=True), required=True) def validate(input_directory, output_directory, asud_filename): try: validation(input_directory, output_directory, asud_filename) @@ -54,8 +54,8 @@ def validate(input_directory, output_directory, asud_filename): sys.exit(1) @cli.command() -@click.argument("input_directory", type=click.Path(exists=True)) -@click.argument("output_directory", type=click.Path()) +@click.option("--i", "input_directory", type=click.Path(exists=True), required=True) +@click.option("--o", "output_directory", type=click.Path(), required=True) @click.option("--crs", default="28349", help="Coordinate Reference System (default: EPSG:28349)") def convert(input_directory, output_directory, crs): try: @@ -66,13 +66,13 @@ def convert(input_directory, output_directory, crs): sys.exit(1) @cli.command() -@click.argument("input_directory", type=click.Path(exists=True)) -@click.argument("output_directory", type=click.Path()) -@click.argument("boundary_filename", type=click.Path(exists=True)) -@click.argument("split_filename", type=click.Path(exists=True)) +@click.option("--i", "input_directory", type=click.Path(exists=True), required=True) +@click.option("--o", "output_directory", type=click.Path(), required=True) +@click.option("--b", "boundary_filename", type=click.Path(exists=True), required=True) +@click.option("--s", "split_filename", type=click.Path(exists=True), required=True) @click.option("--mdc", is_flag=True, help="Export to MDC format", default=False) @click.option("--mdch", is_flag=True, help="Export to MDCH format", default=False) -@click.option("-egs", is_flag=True, help="Export to EGS format", default=False) +@click.option("--egs", is_flag=True, help="Export to EGS format", default=False) def export(input_directory, output_directory, boundary_filename, split_filename, mdc, mdch, egs): try: exports(input_directory, output_directory, boundary_filename, split_filename, mdc, mdch, egs) diff --git a/aemworkflow/conversion.py b/aemworkflow/conversion.py index 11e63b8..688e424 100644 --- a/aemworkflow/conversion.py +++ b/aemworkflow/conversion.py @@ -33,182 +33,190 @@ def conversion_zedfix_gmt_to_srt(wrk_dir: str, path_dir: str, ext_file: str, log """ logger_session.info("Running zedfix_gmt_to_srt conversion.") - fm1 = " {:{_f}}" * 7 + " {} {}\n" - fm2 = " {:{_f}}" * 7 + " {} {}\n" - # regex1 = re.compile('\d') # this fails for negative numbers - regex2 = re.compile('[+-]?([0-9]*[.])?[0-9]+') - - srt_dir = Path(wrk_dir) / "SORT" - get_make_srt_dir(srt_dir, logger_session=logger) - - dcols = ("nm", "frame_l", "frame_top", "frame_r", "frame_bot", "t_l", "t_top", "t_r", "t_bot") - exdf = pd.read_csv(ext_file, sep=r'\s+', names=dcols, header=None, index_col=False) - - logger_session.info("Testing GMT for +Z ") - - # pdf_list = [] - pcols = ("nm", "fid", "pix_x", "pix_y", "coordx", "coordy", "col7", "col8", "gl") - for nm in exdf['nm']: - ner = 0 - fidd = 0 - row = exdf.query("nm == @nm") - y_scale = (row['t_bot'].iloc[0] - row['t_top'].iloc[0]) /\ - (row['frame_bot'].iloc[0] - row['frame_top'].iloc[0]) - p_file = Path(path_dir) / f"{nm}.path.txt" - tdf = pd.read_csv(p_file, sep=r'\s+', names=pcols, header=None, index_col=False) - # pdf_list.append(tdf) - frst = tdf["fid"].iloc[0] - 1 - last = tdf["fid"].iloc[-1] - 1 - - gmt = Path(wrk_dir) / 'interp' / f"{nm}_interp.gmt" - - with open(gmt, "r") as fin: - lin_lst = fin.readlines() - logger_session.info(f"{nm}_interp.gmt successfully read.") - lines = lin_lst.copy() - - with open(Path(srt_dir) / f"{nm}zf.gmtf", "w") as fou: - while lines: - try: - line = lines.pop(0).strip() - except IndexError: - logger_session.info(f"{nm}.gmt processed") - break - # if not regex1.match(line[0]) and ("@D" in line): - if not regex2.match(line.split()[0]) and ("@D" in line): - with open(srt_dir / "met.bdf", mode="a") as fou1: - fou1.write(f"{gmt.name}|{fidd}|{line}\n") - fidd += 1 - fou.write(f"# {line.split('|')[1]}\n") - fou.write(f"{line}\n") - srt_file = srt_dir / f"{nm}_{line.split('|')[1]}.srt" - with open(srt_file, mode="a") as fou3: - fou3.write(">\n") - fou3.write(f"{line}\n") - # elif not regex1.match(line[0]): - elif not regex2.match(line.split()[0]): - if ">" not in line: - with open(srt_dir / f"{nm}_hdr.hdr", mode="a") as fou2: - fou2.write(f"{line}\n") - idd = 1 - fou.write(f"{line}\n") - else: - col_1, col_2 = [float(_l) for _l in line.split()[:2]] - x, y, t = interpol(col_1, frst, last, tdf) - dpth = ((col_2 - row['frame_top'].iloc[0]) * y_scale) + row['t_top'].iloc[0] - if t <= dpth: - # first_col = t - dpth - nyp = col_2 + (t - dpth) / y_scale - fou.write(fm1.format(col_1, nyp, x, y, t, t, 0, idd, fidd - 1, _f=".6f")) + try: + fm1 = " {:{_f}}" * 7 + " {} {}\n" + fm2 = " {:{_f}}" * 7 + " {} {}\n" + # regex1 = re.compile('\d') # this fails for negative numbers + regex2 = re.compile('[+-]?([0-9]*[.])?[0-9]+') + + srt_dir = Path(wrk_dir) / "SORT" + get_make_srt_dir(srt_dir, logger_session=logger) + + dcols = ("nm", "frame_l", "frame_top", "frame_r", "frame_bot", "t_l", "t_top", "t_r", "t_bot") + exdf = pd.read_csv(ext_file, sep=r'\s+', names=dcols, header=None, index_col=False) + + logger_session.info("Testing GMT for +Z ") + + # pdf_list = [] + pcols = ("nm", "fid", "pix_x", "pix_y", "coordx", "coordy", "col7", "col8", "gl") + for nm in exdf['nm']: + ner = 0 + fidd = 0 + row = exdf.query("nm == @nm") + y_scale = (row['t_bot'].iloc[0] - row['t_top'].iloc[0]) /\ + (row['frame_bot'].iloc[0] - row['frame_top'].iloc[0]) + p_file = Path(path_dir) / f"{nm}.path.txt" + tdf = pd.read_csv(p_file, sep=r'\s+', names=pcols, header=None, index_col=False) + # pdf_list.append(tdf) + frst = tdf["fid"].iloc[0] - 1 + last = tdf["fid"].iloc[-1] - 1 + + gmt = Path(wrk_dir) / 'interp' / f"{nm}_interp.gmt" + + with open(gmt, "r") as fin: + lin_lst = fin.readlines() + logger_session.info(f"{nm}_interp.gmt successfully read.") + lines = lin_lst.copy() + + with open(Path(srt_dir) / f"{nm}zf.gmtf", "w") as fou: + while lines: + try: + line = lines.pop(0).strip() + except IndexError: + logger_session.info(f"{nm}.gmt processed") + break + # if not regex1.match(line[0]) and ("@D" in line): + if not regex2.match(line.split()[0]) and ("@D" in line): + with open(srt_dir / "met.bdf", mode="a") as fou1: + fou1.write(f"{gmt.name}|{fidd}|{line}\n") + fidd += 1 + fou.write(f"# {line.split('|')[1]}\n") + fou.write(f"{line}\n") + srt_file = srt_dir / f"{nm}_{line.split('|')[1]}.srt" with open(srt_file, mode="a") as fou3: - fou3.write(fm1.format(col_1, nyp, x, y, t, t, 0, idd, fidd - 1, _f=".6f")) - # fou3.write(fm1.format(col_1, nyp, x, y, t, t, 0, idd, _f=".6f")) - - ner += 1 + fou3.write(">\n") + fou3.write(f"{line}\n") + # elif not regex1.match(line[0]): + elif not regex2.match(line.split()[0]): + if ">" not in line: + with open(srt_dir / f"{nm}_hdr.hdr", mode="a") as fou2: + fou2.write(f"{line}\n") + idd = 1 + fou.write(f"{line}\n") else: - fou.write(fm2.format(col_1, col_2, x, y, dpth, t, t - dpth, idd, fidd - 1, _f=".6f")) - with open(srt_file, mode="a") as fou3: - fou3.write(fm2.format(col_1, col_2, x, y, dpth, t, t - dpth, idd, fidd - 1, _f=".6f")) - # fou3.write(fm2.format(col_1, col_2, x, y, dpth, t, t - dpth, idd, _f=".6f")) - idd += 1 - - fou.write(">\n") - logger_session.info(f"** Error count {ner} **\n") - # logger_session.info("** See z_err.log **\n") - fou.write("# @D0|DNDUTL|||||||||||||||||||||MAL|\n") - for i in range(frst, last + 1): - tmp = f"{-(row['t_top'].iloc[0] - tdf['gl'].iloc[i]) / y_scale: .6f}"\ - .rstrip('0').rstrip('.') - fou.write(f"{i} {tmp}\n") - fou.write(">\n") - - logger_session.info("Completed zedfix_gmt_to_srt conversion.") - - return exdf['nm'].tolist() + col_1, col_2 = [float(_l) for _l in line.split()[:2]] + x, y, t = interpol(col_1, frst, last, tdf) + dpth = ((col_2 - row['frame_top'].iloc[0]) * y_scale) + row['t_top'].iloc[0] + if t <= dpth: + # first_col = t - dpth + nyp = col_2 + (t - dpth) / y_scale + fou.write(fm1.format(col_1, nyp, x, y, t, t, 0, idd, fidd - 1, _f=".6f")) + with open(srt_file, mode="a") as fou3: + fou3.write(fm1.format(col_1, nyp, x, y, t, t, 0, idd, fidd - 1, _f=".6f")) + # fou3.write(fm1.format(col_1, nyp, x, y, t, t, 0, idd, _f=".6f")) + + ner += 1 + else: + fou.write(fm2.format(col_1, col_2, x, y, dpth, t, t - dpth, idd, fidd - 1, _f=".6f")) + with open(srt_file, mode="a") as fou3: + fou3.write(fm2.format(col_1, col_2, x, y, dpth, t, t - dpth, idd, fidd - 1, _f=".6f")) + # fou3.write(fm2.format(col_1, col_2, x, y, dpth, t, t - dpth, idd, _f=".6f")) + idd += 1 + + fou.write(">\n") + logger_session.info(f"** Error count {ner} **\n") + # logger_session.info("** See z_err.log **\n") + fou.write("# @D0|DNDUTL|||||||||||||||||||||MAL|\n") + for i in range(frst, last + 1): + tmp = f"{-(row['t_top'].iloc[0] - tdf['gl'].iloc[i]) / y_scale: .6f}"\ + .rstrip('0').rstrip('.') + fou.write(f"{i} {tmp}\n") + fou.write(">\n") + + logger_session.info("Completed zedfix_gmt_to_srt conversion.") + + return exdf['nm'].tolist() + except Exception as e: + logger_session.error(f"Error during zedfix_gmt_to_srt conversion: {e}") + return [] def conversion_sort_gmtp_3d(wrk_dir: str, nm_lst: List[int], crs: str, logger_session=logger) -> None: logger_session.info("Running sort_gmtp_3d conversion.") - proj = osr.SpatialReference() - proj.ImportFromEPSG(int(crs)) - result_wkt = proj.ExportToWkt() - result_wkt = result_wkt.replace('"', '\\"') - result_proj = proj.ExportToProj4() - # print(f'epsg resutl is: {result_wkt}') - # print(f'proj string is: {result_proj}') - del proj - - srt_dir = Path(wrk_dir) / "SORT" - get_make_srt_dir(srt_dir, logger_session=logger) - - zfshp_dir = Path(wrk_dir) / "ZF_SHP" - if not Path(zfshp_dir).exists(): - Path(zfshp_dir).mkdir(parents=True, exist_ok=False) - else: - logger_session.warning(f"{zfshp_dir} folder already exists!") - - for nm in nm_lst: - ano_list = sorted(glob.glob(os.path.join(srt_dir, "*Annotations.srt"))) - if ano_list: - _ = [Path(_f).unlink() for _f in ano_list] - - srt_list = sorted(glob.glob(os.path.join(srt_dir, f"{nm}*.srt"))) - hdr = Path(srt_dir / f"{nm}_hdr.hdr") - with open(Path(srt_dir) / f"{nm}.gmtsddd", "w") as fou: - with open(hdr, 'r') as hdr_file: - for line in hdr_file: - fields = line.strip().split() - if len(fields) > 0: - if "@VGMT1" in fields[1]: - fou.write(f"{line}") - elif "@R" in fields[1]: - fou.write(f"{line}") - fou.write(f"# @Je{crs}\n") - fou.write(f'# @Jp"{result_proj}"\n') - fou.write(f'# @Jw"{result_wkt}"\n') - elif "@NId" in fields[1]: - hd3 = line.split("# @NId") - fou.write(f"# @NFID_|Entity{hd3[1]}") - elif "@Tinteger" in fields[1]: - hd4 = line.split("# @Tinteger") - fou.write(f"# @Tdouble|string{hd4[1]}") - elif "FEATURE_DATA" in fields[1]: - fou.write(f"{line}") - - for i, srt in enumerate(srt_list, 1): - vtx = 1 - seg = 0 - with open(srt, 'r') as srt_file: - for line in srt_file: + try: + proj = osr.SpatialReference() + proj.ImportFromEPSG(int(crs)) + result_wkt = proj.ExportToWkt() + result_wkt = result_wkt.replace('"', '\\"') + result_proj = proj.ExportToProj4() + # print(f'epsg resutl is: {result_wkt}') + # print(f'proj string is: {result_proj}') + del proj + + srt_dir = Path(wrk_dir) / "SORT" + get_make_srt_dir(srt_dir, logger_session=logger) + + zfshp_dir = Path(wrk_dir) / "ZF_SHP" + if not Path(zfshp_dir).exists(): + Path(zfshp_dir).mkdir(parents=True, exist_ok=False) + else: + logger_session.warning(f"{zfshp_dir} folder already exists!") + + for nm in nm_lst: + ano_list = sorted(glob.glob(os.path.join(srt_dir, "*Annotations.srt"))) + if ano_list: + _ = [Path(_f).unlink() for _f in ano_list] + + srt_list = sorted(glob.glob(os.path.join(srt_dir, f"{nm}*.srt"))) + hdr = Path(srt_dir / f"{nm}_hdr.hdr") + with open(Path(srt_dir) / f"{nm}.gmtsddd", "w") as fou: + with open(hdr, 'r') as hdr_file: + for line in hdr_file: fields = line.strip().split() if len(fields) > 0: - if fields[0] == ">": - seg += 1 - fou.write(f"{line.strip()}\n") - next_line = next(srt_file).strip() - met = next_line.split("# @D0") - fou.write(f"# @D0|3DPolyline{met[1]}\n") - else: - # fou.write(fields[2], fields[3], fields[4], fields[0], fields[1], fields[5], - # fields[6], fields[7], fields[8], vtx, seg) - fou.write(" ".join([fields[2], fields[3], fields[4], fields[0], - fields[1], fields[5], fields[6], fields[7], - fields[8], str(vtx), str(seg)]) + "\n") - vtx += 1 - - in_gmtf = Path(srt_dir) / f"{nm}zf.gmtf" - out_shp = Path(zfshp_dir) / f"{nm}_zf.shp" - - # ogr2ogr.main(["", "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)]) - cmd = [get_ogr_path(), "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)] - if not validate_file(in_gmtf): - return - run_command(cmd) - - if out_shp.exists(): - logger_session.info(f"{nm}zf.gmtf successfully converted to {nm}_zf.shp") - - logger_session.info("Completed sort_gmtp conversion.") + if "@VGMT1" in fields[1]: + fou.write(f"{line}") + elif "@R" in fields[1]: + fou.write(f"{line}") + fou.write(f"# @Je{crs}\n") + fou.write(f'# @Jp"{result_proj}"\n') + fou.write(f'# @Jw"{result_wkt}"\n') + elif "@NId" in fields[1]: + hd3 = line.split("# @NId") + fou.write(f"# @NFID_|Entity{hd3[1]}") + elif "@Tinteger" in fields[1]: + hd4 = line.split("# @Tinteger") + fou.write(f"# @Tdouble|string{hd4[1]}") + elif "FEATURE_DATA" in fields[1]: + fou.write(f"{line}") + + for i, srt in enumerate(srt_list, 1): + vtx = 1 + seg = 0 + with open(srt, 'r') as srt_file: + for line in srt_file: + fields = line.strip().split() + if len(fields) > 0: + if fields[0] == ">": + seg += 1 + fou.write(f"{line.strip()}\n") + next_line = next(srt_file).strip() + met = next_line.split("# @D0") + fou.write(f"# @D0|3DPolyline{met[1]}\n") + else: + # fou.write(fields[2], fields[3], fields[4], fields[0], fields[1], fields[5], + # fields[6], fields[7], fields[8], vtx, seg) + fou.write(" ".join([fields[2], fields[3], fields[4], fields[0], + fields[1], fields[5], fields[6], fields[7], + fields[8], str(vtx), str(seg)]) + "\n") + vtx += 1 + + in_gmtf = Path(srt_dir) / f"{nm}zf.gmtf" + out_shp = Path(zfshp_dir) / f"{nm}_zf.shp" + + # ogr2ogr.main(["", "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)]) + cmd = [get_ogr_path(), "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)] + if not validate_file(in_gmtf): + return + run_command(cmd) + + if out_shp.exists(): + logger_session.info(f"{nm}zf.gmtf successfully converted to {nm}_zf.shp") + + logger_session.info("Completed sort_gmtp conversion.") + except Exception as e: + logger_session.error(f"Error during sort_gmtp_3d conversion: {e}") + return def conversion_sort_gmtp(wrk_dir: str, nm_lst: List[int], logger_session=logger) -> None: @@ -228,62 +236,66 @@ def conversion_sort_gmtp(wrk_dir: str, nm_lst: List[int], logger_session=logger) """ logger_session.info("Running sort_gmtp conversion.") - srt_dir = Path(wrk_dir) / "SORT" - get_make_srt_dir(srt_dir, logger_session=logger) - - zfshp_dir = Path(wrk_dir) / "ZF_SHP" - if not Path(zfshp_dir).exists(): - Path(zfshp_dir).mkdir(parents=True, exist_ok=False) - else: - logger_session.warning(f"{zfshp_dir} folder already exists!") - - for nm in nm_lst: - seg = 0 - vtx = 1 - fn = 1 - ano_list = sorted(glob.glob(os.path.join(srt_dir, "*Annotations.srt"))) - if ano_list: - _ = [Path(_f).unlink() for _f in ano_list] - srt_list = sorted(glob.glob(os.path.join(srt_dir, f"{nm}*.srt"))) - hdr = Path(srt_dir / f"{nm}_hdr.hdr") - hlines = hdr.read_text().split("\n")[:-1] - with open(Path(srt_dir) / f"{nm}.gmts", "w") as fou: - for hline in hlines: - fou.write(f"{hline}\n") - for i, srt in enumerate(srt_list, 1): - lines = Path(srt).read_text().split("\n") - while lines: - try: - line = lines.pop(0) - except IndexError: - logger_session.info(f"{srt} completed") - break - if line.startswith(">"): - seg += 1 - fou.write(f"{line}\n") - line = lines.pop(0) - fou.write(f"{line}\n") - if fn != i: - seg = 1 - vtx = 1 - fn += 1 - elif line: - fou.write(f"{line} {vtx} {seg}\n") - vtx += 1 - - in_gmtf = Path(srt_dir) / f"{nm}zf.gmtf" - out_shp = Path(zfshp_dir) / f"{nm}_zf.shp" - - # ogr2ogr.main(["", "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)]) - cmd = [get_ogr_path(), "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)] - if not validate_file(in_gmtf): - return - run_command(cmd) - - if out_shp.exists(): - logger_session.info(f"{nm}zf.gmtf successfully converted to {nm}_zf.shp") - - logger_session.info("Completed sort_gmtp conversion.") + try: + srt_dir = Path(wrk_dir) / "SORT" + get_make_srt_dir(srt_dir, logger_session=logger) + + zfshp_dir = Path(wrk_dir) / "ZF_SHP" + if not Path(zfshp_dir).exists(): + Path(zfshp_dir).mkdir(parents=True, exist_ok=False) + else: + logger_session.warning(f"{zfshp_dir} folder already exists!") + + for nm in nm_lst: + seg = 0 + vtx = 1 + fn = 1 + ano_list = sorted(glob.glob(os.path.join(srt_dir, "*Annotations.srt"))) + if ano_list: + _ = [Path(_f).unlink() for _f in ano_list] + srt_list = sorted(glob.glob(os.path.join(srt_dir, f"{nm}*.srt"))) + hdr = Path(srt_dir / f"{nm}_hdr.hdr") + hlines = hdr.read_text().split("\n")[:-1] + with open(Path(srt_dir) / f"{nm}.gmts", "w") as fou: + for hline in hlines: + fou.write(f"{hline}\n") + for i, srt in enumerate(srt_list, 1): + lines = Path(srt).read_text().split("\n") + while lines: + try: + line = lines.pop(0) + except IndexError: + logger_session.info(f"{srt} completed") + break + if line.startswith(">"): + seg += 1 + fou.write(f"{line}\n") + line = lines.pop(0) + fou.write(f"{line}\n") + if fn != i: + seg = 1 + vtx = 1 + fn += 1 + elif line: + fou.write(f"{line} {vtx} {seg}\n") + vtx += 1 + + in_gmtf = Path(srt_dir) / f"{nm}zf.gmtf" + out_shp = Path(zfshp_dir) / f"{nm}_zf.shp" + + # ogr2ogr.main(["", "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)]) + cmd = [get_ogr_path(), "-f", "ESRI Shapefile", str(out_shp), str(in_gmtf)] + if not validate_file(in_gmtf): + return + run_command(cmd) + + if out_shp.exists(): + logger_session.info(f"{nm}zf.gmtf successfully converted to {nm}_zf.shp") + + logger_session.info("Completed sort_gmtp conversion.") + except Exception as e: + logger_session.error(f"Error during sort_gmtp conversion: {e}") + return def interpol(col_1: float, frst: int, last: int, tdf: pd.DataFrame) -> Tuple[float]: @@ -308,53 +320,56 @@ def interpol(col_1: float, frst: int, last: int, tdf: pd.DataFrame) -> Tuple[flo The list of path identifiers read from the first column of the extent file """ - i_last = len(tdf) - cdp = (tdf["fid"] - 1).tolist() - if frst <= col_1 < last: - # interpolate - for idx in cdp: - if cdp[idx] <= col_1 < cdp[idx + 1]: - x1 = tdf['coordx'].iloc[idx] - y1 = tdf['coordy'].iloc[idx] - g1 = tdf['gl'].iloc[idx] - x2 = tdf['coordx'].iloc[idx + 1] - y2 = tdf['coordy'].iloc[idx + 1] - g2 = tdf['gl'].iloc[idx + 1] - l1 = cdp[idx + 1] - cdp[idx] - l2 = col_1 - cdp[idx] - lr = l2 / l1 - x = x1 + (x2 - x1) * lr - y = y1 + (y2 - y1) * lr - t = g1 + (g2 - g1) * lr - elif int(col_1) <= frst: - # extrapolate to the left - x1 = tdf['coordx'].iloc[0] - y1 = tdf['coordy'].iloc[0] - g1 = tdf['gl'].iloc[0] - x2 = tdf['coordx'].iloc[1] - y2 = tdf['coordy'].iloc[1] - g2 = tdf['gl'].iloc[1] - l1 = cdp[1] - cdp[0] - l2 = col_1 - cdp[0] - lr = l2 / l1 - x = x1 + (x2 - x1) * lr - y = y1 + (y2 - y1) * lr - t = g1 + (g2 - g1) * lr - elif col_1 >= last: - # extrapolate to the right - x1 = tdf['coordx'].iloc[i_last - 2] - y1 = tdf['coordy'].iloc[i_last - 2] - g1 = tdf['gl'].iloc[i_last - 2] - x2 = tdf['coordx'].iloc[i_last - 1] - y2 = tdf['coordy'].iloc[i_last - 1] - g2 = tdf['gl'].iloc[i_last - 1] - l1 = cdp[i_last - 1] - cdp[i_last - 2] - l2 = col_1 - cdp[i_last - 1] - lr = l2 / l1 - x = x2 + (x2 - x1) * lr - y = y2 + (y2 - y1) * lr - t = g2 + (g2 - g1) * lr - return x, y, t + try: + i_last = len(tdf) + cdp = (tdf["fid"] - 1).tolist() + if frst <= col_1 < last: + # interpolate + for idx in cdp: + if cdp[idx] <= col_1 < cdp[idx + 1]: + x1 = tdf['coordx'].iloc[idx] + y1 = tdf['coordy'].iloc[idx] + g1 = tdf['gl'].iloc[idx] + x2 = tdf['coordx'].iloc[idx + 1] + y2 = tdf['coordy'].iloc[idx + 1] + g2 = tdf['gl'].iloc[idx + 1] + l1 = cdp[idx + 1] - cdp[idx] + l2 = col_1 - cdp[idx] + lr = l2 / l1 + x = x1 + (x2 - x1) * lr + y = y1 + (y2 - y1) * lr + t = g1 + (g2 - g1) * lr + elif int(col_1) <= frst: + # extrapolate to the left + x1 = tdf['coordx'].iloc[0] + y1 = tdf['coordy'].iloc[0] + g1 = tdf['gl'].iloc[0] + x2 = tdf['coordx'].iloc[1] + y2 = tdf['coordy'].iloc[1] + g2 = tdf['gl'].iloc[1] + l1 = cdp[1] - cdp[0] + l2 = col_1 - cdp[0] + lr = l2 / l1 + x = x1 + (x2 - x1) * lr + y = y1 + (y2 - y1) * lr + t = g1 + (g2 - g1) * lr + elif col_1 >= last: + # extrapolate to the right + x1 = tdf['coordx'].iloc[i_last - 2] + y1 = tdf['coordy'].iloc[i_last - 2] + g1 = tdf['gl'].iloc[i_last - 2] + x2 = tdf['coordx'].iloc[i_last - 1] + y2 = tdf['coordy'].iloc[i_last - 1] + g2 = tdf['gl'].iloc[i_last - 1] + l1 = cdp[i_last - 1] - cdp[i_last - 2] + l2 = col_1 - cdp[i_last - 1] + lr = l2 / l1 + x = x2 + (x2 - x1) * lr + y = y2 + (y2 - y1) * lr + t = g2 + (g2 - g1) * lr + return x, y, t + except Exception as e: + logger.error(f"Error during interpolation: {e}") def main(input_directory: str, output_directory: str, crs=28349) -> None: diff --git a/aemworkflow/exports.py b/aemworkflow/exports.py index 4d865b4..ecf7a70 100644 --- a/aemworkflow/exports.py +++ b/aemworkflow/exports.py @@ -14,39 +14,42 @@ def gmtsddd_to_egs(wrk_dir: str, alt_colors: str, nm_list: List[int]) -> None: un = {} # Open the CSV file for writing - with open(os.path.join(wrk_dir, 'SORT', 'output.egs'), 'w', newline='') as csvfile: - csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_NONE, quotechar=None, escapechar='\\') - # Write the header to stderr and the CSV file - sys.stderr.write("Export EGGS CSV\n") - csvwriter.writerow(["Vertex", "SegmentID", "X", "Y", "ELEVATION", "PixelX", "PixelY", "AusAEM_DEM", "DEPTH", - "Type", "OverAge", "UnderAge", "BoundConf", "ContactTyp", "BasisOfInt", "OvrStrtUnt", - "OvrStratNo", "OvrConf", "UndStrtUnt", "UndStratNo", "UndConf", "WithinStrt", - "WithinStNo", "WithinConf", "HydStrtType", "HydStrConf", "BOMNAFUnt", "BOMNAFNo", - "InterpRef", "Comment", "Annotation", "NewObs", "Operator", "Date", "SURVEY_LINE"]) - - # Read the input file - with open(alt_colors, 'r') as prn_file: - for line in prn_file: - # parts = line.strip().split(',') - parts = re.split(r'\s{2,}', line) - ov[parts[0]] = ' ' if len(parts) < 2 else parts[1] - un[parts[0]] = ' ' if len(parts) < 3 else parts[2] - - for filename in nm_list: - with open(os.path.join(wrk_dir, 'SORT', f'{filename}.gmtsddd'), 'r') as file: - for line in file: - # Process the subsequent files - if line.startswith('# @D0'): - # Handle lines starting with '#' - # seg += 1 - parts = line.strip().split('|') - # l = parts[3].split('_') - met = [parts[2]] + [ov.get(parts[3], '')] + [un.get(parts[3], '')] + parts[3:25] - elif line[0].isdigit(): - # Handle lines starting with a digit - parts = line.strip().split(' ') - row_to_write = parts[9:10] + parts[8:9] + parts[0:7] + met + [filename] - csvwriter.writerow(row_to_write) + try: + with open(os.path.join(wrk_dir, 'SORT', 'output.egs'), 'w', newline='') as csvfile: + csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_NONE, quotechar=None, escapechar='\\') + # Write the header to stderr and the CSV file + sys.stderr.write("Export EGGS CSV\n") + csvwriter.writerow(["Vertex", "SegmentID", "X", "Y", "ELEVATION", "PixelX", "PixelY", "AusAEM_DEM", "DEPTH", + "Type", "OverAge", "UnderAge", "BoundConf", "ContactTyp", "BasisOfInt", "OvrStrtUnt", + "OvrStratNo", "OvrConf", "UndStrtUnt", "UndStratNo", "UndConf", "WithinStrt", + "WithinStNo", "WithinConf", "HydStrtType", "HydStrConf", "BOMNAFUnt", "BOMNAFNo", + "InterpRef", "Comment", "Annotation", "NewObs", "Operator", "Date", "SURVEY_LINE"]) + + # Read the input file + with open(alt_colors, 'r') as prn_file: + for line in prn_file: + # parts = line.strip().split(',') + parts = re.split(r'\s{2,}', line) + ov[parts[0]] = ' ' if len(parts) < 2 else parts[1] + un[parts[0]] = ' ' if len(parts) < 3 else parts[2] + + for filename in nm_list: + with open(os.path.join(wrk_dir, 'SORT', f'{filename}.gmtsddd'), 'r') as file: + for line in file: + # Process the subsequent files + if line.startswith('# @D0'): + # Handle lines starting with '#' + # seg += 1 + parts = line.strip().split('|') + # l = parts[3].split('_') + met = [parts[2]] + [ov.get(parts[3], '')] + [un.get(parts[3], '')] + parts[3:25] + elif line[0].isdigit(): + # Handle lines starting with a digit + parts = line.strip().split(' ') + row_to_write = parts[9:10] + parts[8:9] + parts[0:7] + met + [filename] + csvwriter.writerow(row_to_write) + except Exception as e: + logger.error(f"Error during gmtsddd_to_egs conversion: {e}") def gmtsddd_to_mdc(wrk_dir: str, colors: str, nm_list: List[int]) -> None: @@ -55,100 +58,103 @@ def gmtsddd_to_mdc(wrk_dir: str, colors: str, nm_list: List[int]) -> None: b = {} # Open the CSV file for writing - with open(os.path.join(wrk_dir, 'SORT', 'output.mdc'), 'w', newline='') as csvfile: - csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_NONE, quotechar=None, escapechar='\\') - # Write the header to stderr and the CSV file - # sys.stderr.write("Export EGGS CSV\n") - # csvwriter.writerow(["Vertex", "SegmentID", "X", "Y", "ELEVATION", "PixelX", - # "PixelY", "AusAEM_DEM", "DEPTH", "Type", "OverAge", "UnderAge", "BoundConf", - # "ContactTyp", "BasisOfInt", "OvrStrtUnt", "OvrStratNo", "OvrConf", "UndStrtUnt", - # "UndStratNo", "UndConf", "WithinStrt", "WithinStNo", "WithinConf", "HydStrtType", - # "HydStrConf", "BOMNAFUnt", "BOMNAFNo", "InterpRef", "Comment", "Annotation", "NewObs", - # "Operator", "Date", "SURVEY_LINE"]) - - # Read the input file - with open(colors, 'r') as prn_file: - prn_file.readline() - for line in prn_file: - # data = line.strip().split() - data = re.split(r'\s{2,}', line) - if len(data) > 4: - r[data[0]] = float(data[1]) - g[data[0]] = float(data[2]) - b[data[0]] = float(data[3]) - - for filename in nm_list: - with open(os.path.join(wrk_dir, 'SORT', f'{filename}.gmtsddd'), 'r') as file: - for line in file: - if line.startswith("# @D0"): - filen = [filename, ''] # filename.split(".") - line = line.strip().split("|") - second_line = file.readline().strip().split() - segn = second_line[8] - - csvwriter.writerow(["GOCAD PLine 1"]) - csvwriter.writerow(["HEADER {"]) - csvwriter.writerow([f"name:{filen[0]}_{segn}_{line[2]}"]) - csvwriter.writerow(["*atoms:false"]) - csvwriter.writerow(["*line*color:%f %f %f 1" % (r[line[2]] / 256, - g[line[2]] / 256, - b[line[2]] / 256)]) - csvwriter.writerow(["use_feature_color: false"]) - csvwriter.writerow(["width:5"]) - csvwriter.writerow([f"*metadata*Line: {filen[0]}"]) - csvwriter.writerow([f"*metadata*Type: {line[2]}"]) - csvwriter.writerow([f"*metadata*BoundaryNm: {line[3]}"]) - csvwriter.writerow([f"*metadata*BoundConf: {line[4]}"]) - csvwriter.writerow([f"*metadata*BasisOfInt: {line[5]}"]) - csvwriter.writerow([f"*metadata*OvrStrtUnt: {line[6]}"]) - csvwriter.writerow([f"*metadata*OvrStrtCod: {line[7]}"]) - csvwriter.writerow([f"*metadata*OvrConf: {line[8]}"]) - csvwriter.writerow([f"*metadata*UndStrtUnt: {line[9]}"]) - csvwriter.writerow([f"*metadata*UndStrtCod: {line[10]}"]) - csvwriter.writerow([f"*metadata*UndConf: {line[11]}"]) - csvwriter.writerow([f"*metadata*WithinType: {line[12]}"]) - csvwriter.writerow([f"*metadata*WithinStrt: {line[13]}"]) - csvwriter.writerow([f"*metadata*WithinStNo: {line[14]}"]) - csvwriter.writerow([f"*metadata*WithinConf: {line[15]}"]) - csvwriter.writerow([f"*metadata*InterpRef: {line[16]}"]) - csvwriter.writerow([f"*metadata*Comment: {line[17]}"]) - csvwriter.writerow([f"*metadata*Annotation: {line[18]}"]) - csvwriter.writerow([f"*metadata*NewObs: {line[19]}"]) - csvwriter.writerow([f"*metadata*Operator: {line[20]}"]) - csvwriter.writerow(["*metadata*Organization: Geoscience Australia"]) - csvwriter.writerow(["}"]) - csvwriter.writerow(["PROPERTIES px py gl depth"]) - - # Coordinate reference system - csvwriter.writerow(["GOCAD_ORIGINAL_COORDINATE_SYSTEM"]) - csvwriter.writerow(["NAME \" gocad Local\""]) - csvwriter.writerow(["PROJECTION \" GDA94 / MGA zone 53\""]) - csvwriter.writerow(["DATUM \" Mean Sea Level\""]) - csvwriter.writerow(["AXIS_NAME X Y Z"]) - csvwriter.writerow(["AXIS_UNIT m m m"]) - csvwriter.writerow(["ZPOSITIVE Elevation"]) - csvwriter.writerow(["END_ORIGINAL_COORDINATE_SYSTEM"]) - - # Feature class used to group section components (AEM section) - csvwriter.writerow([f"GEOLOGICAL_FEATURE {filen[0]}"]) - csvwriter.writerow(["ILINE"]) - - line = second_line - first = last = int(line[9]) - while True: - last = int(line[9]) - csvwriter.writerow([f"PVRTX {int(line[9])} {float(line[0]):.1f} " - f"{float(line[1]):.1f} {float(line[2]):.1f} " - f"{float(line[3])} {float(line[4])} " - f"{float(line[5]):.1f} {float(line[6]):.1f}"]) - line = file.readline().strip().split() - if not line or not line[0].replace('.', '').isdigit(): - break - - for i in range(first, last): - csvwriter.writerow([f"seg {i} {i + 1}"]) - - csvwriter.writerow(["END"]) + try: + with open(os.path.join(wrk_dir, 'SORT', 'output.mdc'), 'w', newline='') as csvfile: + csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_NONE, quotechar=None, escapechar='\\') + # Write the header to stderr and the CSV file + # sys.stderr.write("Export EGGS CSV\n") + # csvwriter.writerow(["Vertex", "SegmentID", "X", "Y", "ELEVATION", "PixelX", + # "PixelY", "AusAEM_DEM", "DEPTH", "Type", "OverAge", "UnderAge", "BoundConf", + # "ContactTyp", "BasisOfInt", "OvrStrtUnt", "OvrStratNo", "OvrConf", "UndStrtUnt", + # "UndStratNo", "UndConf", "WithinStrt", "WithinStNo", "WithinConf", "HydStrtType", + # "HydStrConf", "BOMNAFUnt", "BOMNAFNo", "InterpRef", "Comment", "Annotation", "NewObs", + # "Operator", "Date", "SURVEY_LINE"]) + + # Read the input file + with open(colors, 'r') as prn_file: + prn_file.readline() + for line in prn_file: + # data = line.strip().split() + data = re.split(r'\s{2,}', line) + if len(data) > 4: + r[data[0]] = float(data[1]) + g[data[0]] = float(data[2]) + b[data[0]] = float(data[3]) + + for filename in nm_list: + with open(os.path.join(wrk_dir, 'SORT', f'{filename}.gmtsddd'), 'r') as file: + for line in file: + if line.startswith("# @D0"): + filen = [filename, ''] # filename.split(".") + line = line.strip().split("|") + second_line = file.readline().strip().split() + segn = second_line[8] + + csvwriter.writerow(["GOCAD PLine 1"]) + csvwriter.writerow(["HEADER {"]) + csvwriter.writerow([f"name:{filen[0]}_{segn}_{line[2]}"]) + csvwriter.writerow(["*atoms:false"]) + csvwriter.writerow(["*line*color:%f %f %f 1" % (r[line[2]] / 256, + g[line[2]] / 256, + b[line[2]] / 256)]) + csvwriter.writerow(["use_feature_color: false"]) + csvwriter.writerow(["width:5"]) + csvwriter.writerow([f"*metadata*Line: {filen[0]}"]) + csvwriter.writerow([f"*metadata*Type: {line[2]}"]) + csvwriter.writerow([f"*metadata*BoundaryNm: {line[3]}"]) + csvwriter.writerow([f"*metadata*BoundConf: {line[4]}"]) + csvwriter.writerow([f"*metadata*BasisOfInt: {line[5]}"]) + csvwriter.writerow([f"*metadata*OvrStrtUnt: {line[6]}"]) + csvwriter.writerow([f"*metadata*OvrStrtCod: {line[7]}"]) + csvwriter.writerow([f"*metadata*OvrConf: {line[8]}"]) + csvwriter.writerow([f"*metadata*UndStrtUnt: {line[9]}"]) + csvwriter.writerow([f"*metadata*UndStrtCod: {line[10]}"]) + csvwriter.writerow([f"*metadata*UndConf: {line[11]}"]) + csvwriter.writerow([f"*metadata*WithinType: {line[12]}"]) + csvwriter.writerow([f"*metadata*WithinStrt: {line[13]}"]) + csvwriter.writerow([f"*metadata*WithinStNo: {line[14]}"]) + csvwriter.writerow([f"*metadata*WithinConf: {line[15]}"]) + csvwriter.writerow([f"*metadata*InterpRef: {line[16]}"]) + csvwriter.writerow([f"*metadata*Comment: {line[17]}"]) + csvwriter.writerow([f"*metadata*Annotation: {line[18]}"]) + csvwriter.writerow([f"*metadata*NewObs: {line[19]}"]) + csvwriter.writerow([f"*metadata*Operator: {line[20]}"]) + csvwriter.writerow(["*metadata*Organization: Geoscience Australia"]) + csvwriter.writerow(["}"]) + csvwriter.writerow(["PROPERTIES px py gl depth"]) + + # Coordinate reference system + csvwriter.writerow(["GOCAD_ORIGINAL_COORDINATE_SYSTEM"]) + csvwriter.writerow(["NAME \" gocad Local\""]) + csvwriter.writerow(["PROJECTION \" GDA94 / MGA zone 53\""]) + csvwriter.writerow(["DATUM \" Mean Sea Level\""]) + csvwriter.writerow(["AXIS_NAME X Y Z"]) + csvwriter.writerow(["AXIS_UNIT m m m"]) + csvwriter.writerow(["ZPOSITIVE Elevation"]) + csvwriter.writerow(["END_ORIGINAL_COORDINATE_SYSTEM"]) + + # Feature class used to group section components (AEM section) + csvwriter.writerow([f"GEOLOGICAL_FEATURE {filen[0]}"]) + csvwriter.writerow(["ILINE"]) + + line = second_line + first = last = int(line[9]) + while True: + last = int(line[9]) + csvwriter.writerow([f"PVRTX {int(line[9])} {float(line[0]):.1f} " + f"{float(line[1]):.1f} {float(line[2]):.1f} " + f"{float(line[3])} {float(line[4])} " + f"{float(line[5]):.1f} {float(line[6]):.1f}"]) + line = file.readline().strip().split() + if not line or not line[0].replace('.', '').isdigit(): + break + + for i in range(first, last): + csvwriter.writerow([f"seg {i} {i + 1}"]) + + csvwriter.writerow(["END"]) + except Exception as e: + logger.error(f"Error during gmtsddd_to_mdc conversion: {e}") def gmtsddd_to_mdch(wrk_dir: str, colors: str, nm_list: List[int]) -> None: @@ -157,100 +163,103 @@ def gmtsddd_to_mdch(wrk_dir: str, colors: str, nm_list: List[int]) -> None: b = {} # Open the CSV file for writing - with open(os.path.join(wrk_dir, 'SORT', 'output.mdch'), 'w', newline='') as csvfile: - csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_NONE, quotechar=None, escapechar='\\') - # Write the header to stderr and the CSV file - # sys.stderr.write("Export EGGS CSV\n") - # csvwriter.writerow(["Vertex", "SegmentID", "X", "Y", "ELEVATION", "PixelX", "PixelY", - # "AusAEM_DEM", "DEPTH", "Type", "OverAge", "UnderAge", "BoundConf", "ContactTyp", - # "BasisOfInt", "OvrStrtUnt", "OvrStratNo", "OvrConf", "UndStrtUnt", "UndStratNo", - # "UndConf", "WithinStrt", "WithinStNo", "WithinConf", "HydStrtType", "HydStrConf", - # "BOMNAFUnt", "BOMNAFNo", "InterpRef", "Comment", "Annotation", "NewObs", "Operator", - # "Date", "SURVEY_LINE"]) - - # Read the input file - with open(colors, 'r') as prn_file: - prn_file.readline() - for line in prn_file: - # data = line.strip().split() - data = re.split(r'\s{2,}', line) - if len(data) > 4: - r[data[0]] = float(data[1]) - g[data[0]] = float(data[2]) - b[data[0]] = float(data[3]) - - for filename in nm_list: - with open(os.path.join(wrk_dir, 'SORT', f'{filename}.gmtsddd'), 'r') as file: - for line in file: - if line.startswith("# @D0"): - filen = [filename, 'gmtsddd'] # filename.split(".") - line = line.strip().split("|") - second_line = file.readline().strip().split() - segn = second_line[8] - - csvwriter.writerow(["GOCAD PLine 1"]) - csvwriter.writerow(["HEADER {"]) - csvwriter.writerow([f"name:{filen[0]}_{segn}_{line[2]}"]) - csvwriter.writerow(["*atoms:false"]) - csvwriter.writerow(["*line*color: %f %f %f 1" % (r[line[2]] / 256, - g[line[2]] / 256, - b[line[2]] / 256)]) - csvwriter.writerow(["use_feature_color: false"]) - csvwriter.writerow(["width: 5"]) - csvwriter.writerow([f"*metadata*Line: {filen[0]}"]) - csvwriter.writerow([f"*metadata*Type: {line[2]}"]) - csvwriter.writerow([f"*metadata*BoundaryNm: {line[3]}"]) - csvwriter.writerow([f"*metadata*BoundConf: {line[4]}"]) - csvwriter.writerow([f"*metadata*BasisOfInt: {line[5]}"]) - csvwriter.writerow([f"*metadata*OvrStrtUnt: {line[6]}"]) - csvwriter.writerow([f"*metadata*OvrStrtCod: {line[7]}"]) - csvwriter.writerow([f"*metadata*OvrConf: {line[8]}"]) - csvwriter.writerow([f"*metadata*UndStrtUnt: {line[9]}"]) - csvwriter.writerow([f"*metadata*UndStrtCod: {line[10]}"]) - csvwriter.writerow([f"*metadata*UndConf: {line[11]}"]) - csvwriter.writerow([f"*metadata*WithinType: {line[12]}"]) - csvwriter.writerow([f"*metadata*WithinStrt: {line[13]}"]) - csvwriter.writerow([f"*metadata*WithinStNo: {line[14]}"]) - csvwriter.writerow([f"*metadata*WithinConf: {line[15]}"]) - csvwriter.writerow([f"*metadata*InterpRef: {line[16]}"]) - csvwriter.writerow([f"*metadata*Comment: {line[17]}"]) - csvwriter.writerow([f"*metadata*Annotation: {line[18]}"]) - csvwriter.writerow([f"*metadata*NewObs: {line[19]}"]) - csvwriter.writerow([f"*metadata*Operator: {line[20]}"]) - csvwriter.writerow(["*metadata*Organization: Geoscience Australia"]) - csvwriter.writerow(["}"]) - csvwriter.writerow(["PROPERTIES px py gl depth"]) - - # Coordinate reference system - csvwriter.writerow(["GOCAD_ORIGINAL_COORDINATE_SYSTEM"]) - csvwriter.writerow(["NAME \" gocad Local\""]) - csvwriter.writerow(["PROJECTION \" GDA94 / MGA zone 53\""]) - csvwriter.writerow(["DATUM \" Mean Sea Level\""]) - csvwriter.writerow(["AXIS_NAME X Y Z"]) - csvwriter.writerow(["AXIS_UNIT m m m"]) - csvwriter.writerow(["ZPOSITIVE Elevation"]) - csvwriter.writerow(["END_ORIGINAL_COORDINATE_SYSTEM"]) - - # Feature class used to group section components (AEM section) - csvwriter.writerow([f"GEOLOGICAL_FEATURE {line[2]}"]) - csvwriter.writerow(["ILINE"]) - - line = second_line - first = last = int(line[9]) - while True: - last = int(line[9]) - csvwriter.writerow([f"PVRTX {int(line[9])} {float(line[0]):.1f} " - f"{float(line[1]):.1f} {float(line[2]):.1f} " - f"{float(line[3])} {float(line[4])} " - f"{float(line[5]):.1f} {float(line[6]):.1f}"]) - line = file.readline().strip().split() - if not line or not line[0].replace('.', '').isdigit(): - break - - for i in range(first, last): - csvwriter.writerow([f"seg {i} {i + 1}"]) - - csvwriter.writerow(["END"]) + try: + with open(os.path.join(wrk_dir, 'SORT', 'output.mdch'), 'w', newline='') as csvfile: + csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_NONE, quotechar=None, escapechar='\\') + # Write the header to stderr and the CSV file + # sys.stderr.write("Export EGGS CSV\n") + # csvwriter.writerow(["Vertex", "SegmentID", "X", "Y", "ELEVATION", "PixelX", "PixelY", + # "AusAEM_DEM", "DEPTH", "Type", "OverAge", "UnderAge", "BoundConf", "ContactTyp", + # "BasisOfInt", "OvrStrtUnt", "OvrStratNo", "OvrConf", "UndStrtUnt", "UndStratNo", + # "UndConf", "WithinStrt", "WithinStNo", "WithinConf", "HydStrtType", "HydStrConf", + # "BOMNAFUnt", "BOMNAFNo", "InterpRef", "Comment", "Annotation", "NewObs", "Operator", + # "Date", "SURVEY_LINE"]) + + # Read the input file + with open(colors, 'r') as prn_file: + prn_file.readline() + for line in prn_file: + # data = line.strip().split() + data = re.split(r'\s{2,}', line) + if len(data) > 4: + r[data[0]] = float(data[1]) + g[data[0]] = float(data[2]) + b[data[0]] = float(data[3]) + + for filename in nm_list: + with open(os.path.join(wrk_dir, 'SORT', f'{filename}.gmtsddd'), 'r') as file: + for line in file: + if line.startswith("# @D0"): + filen = [filename, 'gmtsddd'] # filename.split(".") + line = line.strip().split("|") + second_line = file.readline().strip().split() + segn = second_line[8] + + csvwriter.writerow(["GOCAD PLine 1"]) + csvwriter.writerow(["HEADER {"]) + csvwriter.writerow([f"name:{filen[0]}_{segn}_{line[2]}"]) + csvwriter.writerow(["*atoms:false"]) + csvwriter.writerow(["*line*color: %f %f %f 1" % (r[line[2]] / 256, + g[line[2]] / 256, + b[line[2]] / 256)]) + csvwriter.writerow(["use_feature_color: false"]) + csvwriter.writerow(["width: 5"]) + csvwriter.writerow([f"*metadata*Line: {filen[0]}"]) + csvwriter.writerow([f"*metadata*Type: {line[2]}"]) + csvwriter.writerow([f"*metadata*BoundaryNm: {line[3]}"]) + csvwriter.writerow([f"*metadata*BoundConf: {line[4]}"]) + csvwriter.writerow([f"*metadata*BasisOfInt: {line[5]}"]) + csvwriter.writerow([f"*metadata*OvrStrtUnt: {line[6]}"]) + csvwriter.writerow([f"*metadata*OvrStrtCod: {line[7]}"]) + csvwriter.writerow([f"*metadata*OvrConf: {line[8]}"]) + csvwriter.writerow([f"*metadata*UndStrtUnt: {line[9]}"]) + csvwriter.writerow([f"*metadata*UndStrtCod: {line[10]}"]) + csvwriter.writerow([f"*metadata*UndConf: {line[11]}"]) + csvwriter.writerow([f"*metadata*WithinType: {line[12]}"]) + csvwriter.writerow([f"*metadata*WithinStrt: {line[13]}"]) + csvwriter.writerow([f"*metadata*WithinStNo: {line[14]}"]) + csvwriter.writerow([f"*metadata*WithinConf: {line[15]}"]) + csvwriter.writerow([f"*metadata*InterpRef: {line[16]}"]) + csvwriter.writerow([f"*metadata*Comment: {line[17]}"]) + csvwriter.writerow([f"*metadata*Annotation: {line[18]}"]) + csvwriter.writerow([f"*metadata*NewObs: {line[19]}"]) + csvwriter.writerow([f"*metadata*Operator: {line[20]}"]) + csvwriter.writerow(["*metadata*Organization: Geoscience Australia"]) + csvwriter.writerow(["}"]) + csvwriter.writerow(["PROPERTIES px py gl depth"]) + + # Coordinate reference system + csvwriter.writerow(["GOCAD_ORIGINAL_COORDINATE_SYSTEM"]) + csvwriter.writerow(["NAME \" gocad Local\""]) + csvwriter.writerow(["PROJECTION \" GDA94 / MGA zone 53\""]) + csvwriter.writerow(["DATUM \" Mean Sea Level\""]) + csvwriter.writerow(["AXIS_NAME X Y Z"]) + csvwriter.writerow(["AXIS_UNIT m m m"]) + csvwriter.writerow(["ZPOSITIVE Elevation"]) + csvwriter.writerow(["END_ORIGINAL_COORDINATE_SYSTEM"]) + + # Feature class used to group section components (AEM section) + csvwriter.writerow([f"GEOLOGICAL_FEATURE {line[2]}"]) + csvwriter.writerow(["ILINE"]) + + line = second_line + first = last = int(line[9]) + while True: + last = int(line[9]) + csvwriter.writerow([f"PVRTX {int(line[9])} {float(line[0]):.1f} " + f"{float(line[1]):.1f} {float(line[2]):.1f} " + f"{float(line[3])} {float(line[4])} " + f"{float(line[5]):.1f} {float(line[6]):.1f}"]) + line = file.readline().strip().split() + if not line or not line[0].replace('.', '').isdigit(): + break + + for i in range(first, last): + csvwriter.writerow([f"seg {i} {i + 1}"]) + + csvwriter.writerow(["END"]) + except Exception as e: + logger.error(f"Error during gmtsddd_to_mdch conversion: {e}") def gmts_2_egs(wrk_dir: str, alt_colors: str, nm_lst: List[int]) -> None: @@ -268,61 +277,65 @@ def gmts_2_egs(wrk_dir: str, alt_colors: str, nm_lst: List[int]) -> None: The list of path identifiers from the the common extent file """ - srt_dir = Path(wrk_dir) / "SORT" - if not (srt_dir).exists(): - logger.error("SORT folder missing") - sys.exit(0) - - header = "Vertex,SegmentID,X,Y,ELEVATION,PixelX," \ - "PixelY,AusAEM_DEM,DEPTH,Type,OverAge,UnderAge," \ - "BoundConf,ContactTyp,BasisOfInt,OvrStrtUnt," \ - "OvrStratNo,OvrConf,UndStrtUnt,UndStratNo," \ - "UndConf,WithinStrt,WithinStNo,WithinConf," \ - "HydStrtType,HydStrConf,BOMNAFUnt,BOMNAFNo," \ - "InterpRef,Comment,Annotation,NewObs,Operator," \ - "" \ - "Date,SURVEY_LINE\n" - - regex2 = re.compile('[+-]?([0-9]*[.])?[0-9]+') - - cdf = pd.read_csv(alt_colors, sep=r"\s{2,}", header=0, index_col=False, engine="python") - cdf.fillna('', inplace=True) - seg = 0 - - for nm in nm_lst: - gmts = Path(srt_dir) / f"{nm}.gmtsddd" - if not gmts.exists(): - continue - lines = gmts.read_text().split("\n") - with open(Path(srt_dir) / f"{nm}.egs", "w") as fou: - fou.write(header) - while lines: - try: - line = lines.pop(0) - except IndexError: - logger.info(f"{nm}.gmts processed") - break - if "@D" in line: - seg += 1 - m_lst = line.split("|") - gnm = m_lst[2] - row = cdf[cdf["TYPE"] == gnm] - # row = cdf.query("TYPE == @gnm") - met = f"{gnm},{row['OVERAGE'].iloc[0]}," - f"{row['UNDERAGE'].iloc[0]}," - f"{','.join(str(x) for x in m_lst[2:24])}" - line = lines.pop(0) - + try: + srt_dir = Path(wrk_dir) / "SORT" + if not (srt_dir).exists(): + logger.error("SORT folder missing") + sys.exit(0) + + header = "Vertex,SegmentID,X,Y,ELEVATION,PixelX," \ + "PixelY,AusAEM_DEM,DEPTH,Type,OverAge,UnderAge," \ + "BoundConf,ContactTyp,BasisOfInt,OvrStrtUnt," \ + "OvrStratNo,OvrConf,UndStrtUnt,UndStratNo," \ + "UndConf,WithinStrt,WithinStNo,WithinConf," \ + "HydStrtType,HydStrConf,BOMNAFUnt,BOMNAFNo," \ + "InterpRef,Comment,Annotation,NewObs,Operator," \ + "" \ + "Date,SURVEY_LINE\n" + + regex2 = re.compile('[+-]?([0-9]*[.])?[0-9]+') + + cdf = pd.read_csv(alt_colors, sep=r"\s{2,}", header=0, index_col=False, engine="python") + cdf.fillna('', inplace=True) + seg = 0 + + for nm in nm_lst: + gmts = Path(srt_dir) / f"{nm}.gmtsddd" + if not gmts.exists(): + continue + lines = gmts.read_text().split("\n") + with open(Path(srt_dir) / f"{nm}.egs", "w") as fou: + fou.write(header) + while lines: try: - while regex2.match(line.split()[0]): - los = line.split() - fou.write(f"{los[9]},{los[8]},{los[2]}," - f"{los[3]},{los[4]},{los[0]}," - f"{los[1]},{los[5]},{los[6]}," - f"{met},{nm}\n") - line = lines.pop(0) + line = lines.pop(0) except IndexError: - pass + logger.info(f"{nm}.gmts processed") + break + if "@D" in line: + seg += 1 + m_lst = line.split("|") + gnm = m_lst[2] + row = cdf[cdf["TYPE"] == gnm] + # row = cdf.query("TYPE == @gnm") + met = f"{gnm},{row['OVERAGE'].iloc[0]}," + f"{row['UNDERAGE'].iloc[0]}," + f"{','.join(str(x) for x in m_lst[2:24])}" + line = lines.pop(0) + + try: + while regex2.match(line.split()[0]): + los = line.split() + fou.write(f"{los[9]},{los[8]},{los[2]}," + f"{los[3]},{los[4]},{los[0]}," + f"{los[1]},{los[5]},{los[6]}," + f"{met},{nm}\n") + line = lines.pop(0) + except IndexError: + pass + except Exception as e: + logger.error(f"Error during gmts_2_egs conversion: {e}") + sys.exit(1) def gmts_2_mdc(wrk_dir: str, colors: str, nm_lst: List[int]) -> None: @@ -340,97 +353,101 @@ def gmts_2_mdc(wrk_dir: str, colors: str, nm_lst: List[int]) -> None: The list of path identifiers from the the common extent file """ - fsctn = ("GOCAD PLine 1\n" - "HEADER {{\n" - "name:{}_{}_{}\n" - "*atoms:false\n" - "*line*color:{:.6f} {:.6f} {:.6f} 1\n" - "use_feature_color: false\n" - "width:5\n" - "*metadata*Line: {}\n" - "*metadata*Type: {}\n" - "*metadata*BoundaryNm: {}\n" - "*metadata*BoundConf: {}\n" - "*metadata*BasisOfInt: {}\n" - "*metadata*OvrStrtUnt: {}\n" - "*metadata*OvrStrtCod: {}\n" - "*metadata*OvrConf: {}\n" - "*metadata*UndStrtUnt: {}\n" - "*metadata*UndStrtCod: {}\n" - "*metadata*UndConf: {}\n" - "*metadata*WithinType: {}\n" - "*metadata*WithinStrt: {}\n" - "*metadata*WithinStNo: {}\n" - "*metadata*WithinConf: {}\n" - "*metadata*InterpRef: {}\n" - "*metadata*Comment: {}\n" - "*metadata*Annotation: {}\n" - "*metadata*NewObs: {}\n" - "*metadata*Operator: {}\n" - "*metadata*Organization: Geoscience Australia\n" - "}}\n" - "PROPERTIES px py gl depth\n" - "GOCAD_ORIGINAL_COORDINATE_SYSTEM\n" - 'NAME " gocad Local"\n' - 'PROJECTION " GDA94 / MGA zone 53"\n' - 'DATUM " Mean Sea Level"\n' - "AXIS_NAME X Y Z\n" - "AXIS_UNIT m m m\n" - "ZPOSITIVE Elevation\n" - "END_ORIGINAL_COORDINATE_SYSTEM\n" - "GEOLOGICAL_FEATURE {}\n" - "ILINE\n" - ) - fmt = "PVRTX {:0.0f} {:6.1f} {:7.1f} {:.1f} {:.6f} {:.6f} {:.1f} {:.1f}\n" - - srt_dir = Path(wrk_dir) / "SORT" - if not (srt_dir).exists(): - logger.error("SORT folder missing") - sys.exit(0) - - regex2 = re.compile('[+-]?([0-9]*[.])?[0-9]+') - - cdf = pd.read_csv(colors, sep=r"\s{2,}", header=0, index_col=False, engine="python") - cdf.iloc[:, 1:4] /= 256.0 - - for nm in nm_lst: - gmts = Path(srt_dir) / f"{nm}.gmtsddd" - if not gmts.exists(): - continue - lines = gmts.read_text().split("\n") - with open(Path(srt_dir) / f"{nm}.mdc", "w") as fou: - while lines: - try: - line = lines.pop(0) - except IndexError: - logger.info(f"{nm}.gmts processed") - break - if "@D" in line: - met = line.split("|") - gname = met[2] - row = cdf[cdf["Feature classes"] == gname] - # row = cdf.query("`Feature classes` == @gname") - line = lines.pop(0) - segn, frst = line.split()[8: 10] - frst = int(frst) - # segn = int(segn) - fou.write(fsctn.format(nm, segn, met[2], - row['Red'].iloc[0], row['Green'].iloc[0], row['Blue'].iloc[0], - nm, - *met[2:21], - nm - )) + try: + fsctn = ("GOCAD PLine 1\n" + "HEADER {{\n" + "name:{}_{}_{}\n" + "*atoms:false\n" + "*line*color:{:.6f} {:.6f} {:.6f} 1\n" + "use_feature_color: false\n" + "width:5\n" + "*metadata*Line: {}\n" + "*metadata*Type: {}\n" + "*metadata*BoundaryNm: {}\n" + "*metadata*BoundConf: {}\n" + "*metadata*BasisOfInt: {}\n" + "*metadata*OvrStrtUnt: {}\n" + "*metadata*OvrStrtCod: {}\n" + "*metadata*OvrConf: {}\n" + "*metadata*UndStrtUnt: {}\n" + "*metadata*UndStrtCod: {}\n" + "*metadata*UndConf: {}\n" + "*metadata*WithinType: {}\n" + "*metadata*WithinStrt: {}\n" + "*metadata*WithinStNo: {}\n" + "*metadata*WithinConf: {}\n" + "*metadata*InterpRef: {}\n" + "*metadata*Comment: {}\n" + "*metadata*Annotation: {}\n" + "*metadata*NewObs: {}\n" + "*metadata*Operator: {}\n" + "*metadata*Organization: Geoscience Australia\n" + "}}\n" + "PROPERTIES px py gl depth\n" + "GOCAD_ORIGINAL_COORDINATE_SYSTEM\n" + 'NAME " gocad Local"\n' + 'PROJECTION " GDA94 / MGA zone 53"\n' + 'DATUM " Mean Sea Level"\n' + "AXIS_NAME X Y Z\n" + "AXIS_UNIT m m m\n" + "ZPOSITIVE Elevation\n" + "END_ORIGINAL_COORDINATE_SYSTEM\n" + "GEOLOGICAL_FEATURE {}\n" + "ILINE\n" + ) + fmt = "PVRTX {:0.0f} {:6.1f} {:7.1f} {:.1f} {:.6f} {:.6f} {:.1f} {:.1f}\n" + + srt_dir = Path(wrk_dir) / "SORT" + if not (srt_dir).exists(): + logger.error("SORT folder missing") + sys.exit(0) + + regex2 = re.compile('[+-]?([0-9]*[.])?[0-9]+') + + cdf = pd.read_csv(colors, sep=r"\s{2,}", header=0, index_col=False, engine="python") + cdf.iloc[:, 1:4] /= 256.0 + + for nm in nm_lst: + gmts = Path(srt_dir) / f"{nm}.gmtsddd" + if not gmts.exists(): + continue + lines = gmts.read_text().split("\n") + with open(Path(srt_dir) / f"{nm}.mdc", "w") as fou: + while lines: try: - while regex2.match(line.split()[0]): - los = [int(_l) if i == 7 else float(_l) for i, _l in enumerate(line.split())] - fou.write(fmt.format(los[9], los[2], los[3], los[4], los[0], los[1], los[5], los[6])) - last = int(los[9]) - line = lines.pop(0) + line = lines.pop(0) except IndexError: - pass - for i in range(frst, last): - fou.write(f"seg {i} {i + 1}\n") - fou.write("END\n") + logger.info(f"{nm}.gmts processed") + break + if "@D" in line: + met = line.split("|") + gname = met[2] + row = cdf[cdf["Feature classes"] == gname] + # row = cdf.query("`Feature classes` == @gname") + line = lines.pop(0) + segn, frst = line.split()[8: 10] + frst = int(frst) + # segn = int(segn) + fou.write(fsctn.format(nm, segn, met[2], + row['Red'].iloc[0], row['Green'].iloc[0], row['Blue'].iloc[0], + nm, + *met[2:21], + nm + )) + try: + while regex2.match(line.split()[0]): + los = [int(_l) if i == 7 else float(_l) for i, _l in enumerate(line.split())] + fou.write(fmt.format(los[9], los[2], los[3], los[4], los[0], los[1], los[5], los[6])) + last = int(los[9]) + line = lines.pop(0) + except IndexError: + pass + for i in range(frst, last): + fou.write(f"seg {i} {i + 1}\n") + fou.write("END\n") + except Exception as e: + logger.error(f"Error during gmts_2_mdc conversion: {e}") + sys.exit(1) def main(input_directory: str, output_directory: str, boundary: str, split: str, diff --git a/aemworkflow/interpretation.py b/aemworkflow/interpretation.py index 223b1d7..5bf4f21 100644 --- a/aemworkflow/interpretation.py +++ b/aemworkflow/interpretation.py @@ -18,17 +18,20 @@ def active_gmt_metadata_to_bdf(gmt_file_path, bdf_file_path, mode): - with open(bdf_file_path, mode) as out_bdf_file: - with open(gmt_file_path) as in_gmt_file: - counter = 0 - for line in in_gmt_file: - if '@D' in line: - out_bdf_file.write(f"{Path(gmt_file_path).name}|{counter}|{line.strip()}\n") - counter += 1 + try: + with open(bdf_file_path, mode) as out_bdf_file: + with open(gmt_file_path) as in_gmt_file: + counter = 0 + for line in in_gmt_file: + if '@D' in line: + out_bdf_file.write(f"{Path(gmt_file_path).name}|{counter}|{line.strip()}\n") + counter += 1 + except Exception as e: + print(f"Error processing GMT metadata: {e}", file=sys.stderr) + sys.exit(1) def active_shp_to_gmt(shp_file_path, gmt_file_path): - # ogr2ogr.main(["", "-f", "GMT", gmt_file_path, shp_file_path]) cmd = [get_ogr_path(), "-f", "GMT", gmt_file_path, shp_file_path] if not validate_file(shp_file_path): return @@ -38,41 +41,45 @@ def active_shp_to_gmt(shp_file_path, gmt_file_path): def active_extent_control_file(extent_file_path, path_file_path, output_file_path, out_active_extent_path, crs, gis, mode): - proj = osr.SpatialReference() - proj.ImportFromEPSG(int(crs)) - result_wkt = proj.ExportToWkt() - result_wkt = result_wkt.replace('"', '\\"') - result_proj = proj.ExportToProj4() - # print(f'epsg resutl is: {result_wkt}') - # print(f'proj string is: {result_proj}') - del proj - - with open(out_active_extent_path, mode) as out_active_ext_file: - with open(extent_file_path) as extent_file: - extent_line = extent_file.readline().strip() - out_active_ext_file.write(f'{extent_line}\n') - - with open(output_file_path, mode) as out_file: - if mode == 'w': - out_file.write("# @VGMT1.0 @GLINESTRING\n") - out_file.write(f'# @Jp"{result_proj}"\n') - out_file.write(f'# @Jw"{result_wkt}"\n') - out_file.write("# @Nlinenum\n") - out_file.write("# @Tinteger\n") - out_file.write("# FEATURE_DATA\n") - - line_name = None - - with open(path_file_path) as path_file: - for path_line in path_file: - path_line = path_line.strip().split() - - if line_name != path_line[0]: - line_name = path_line[0] - out_file.write(">\n") - out_file.write(f"# @D{line_name}\n") - - out_file.write(f"{path_line[4]} {path_line[5]}\n") + try: + proj = osr.SpatialReference() + proj.ImportFromEPSG(int(crs)) + result_wkt = proj.ExportToWkt() + result_wkt = result_wkt.replace('"', '\\"') + result_proj = proj.ExportToProj4() + # print(f'epsg resutl is: {result_wkt}') + # print(f'proj string is: {result_proj}') + del proj + + with open(out_active_extent_path, mode) as out_active_ext_file: + with open(extent_file_path) as extent_file: + extent_line = extent_file.readline().strip() + out_active_ext_file.write(f'{extent_line}\n') + + with open(output_file_path, mode) as out_file: + if mode == 'w': + out_file.write("# @VGMT1.0 @GLINESTRING\n") + out_file.write(f'# @Jp"{result_proj}"\n') + out_file.write(f'# @Jw"{result_wkt}"\n') + out_file.write("# @Nlinenum\n") + out_file.write("# @Tinteger\n") + out_file.write("# FEATURE_DATA\n") + + line_name = None + + with open(path_file_path) as path_file: + for path_line in path_file: + path_line = path_line.strip().split() + + if line_name != path_line[0]: + line_name = path_line[0] + out_file.write(">\n") + out_file.write(f"# @D{line_name}\n") + + out_file.write(f"{path_line[4]} {path_line[5]}\n") + except Exception as e: + print(f"Error processing extent or path file: {e}", file=sys.stderr) + sys.exit(1) def main(input_directory, output_directory, crs=28349, gis="esri_arcmap_0.5", lines=10, lines_increment=30): diff --git a/aemworkflow/pre_interpretation.py b/aemworkflow/pre_interpretation.py index c9591f5..bf94a98 100644 --- a/aemworkflow/pre_interpretation.py +++ b/aemworkflow/pre_interpretation.py @@ -14,120 +14,132 @@ def all_lines(path_file_path, output_file_path, crs, gis, mode): - proj = osr.SpatialReference() - proj.ImportFromEPSG(int(crs)) - result_wkt = proj.ExportToWkt() - result_wkt = result_wkt.replace('"', '\\"') - result_proj = proj.ExportToProj4() - # print(f'epsg resutl is: {result_wkt}') - # print(f'proj string is: {result_proj}') - del proj - - with open(output_file_path, mode) as out_file: - if mode == 'w': - out_file.write("# @VGMT1.0 @GLINESTRING\n") - out_file.write(f'# @Jp"{result_proj}"\n') - out_file.write(f'# @Jw"{result_wkt}"\n') - out_file.write("# @Nlinenum|flightnum|date|Survey|Company|Status\n") - out_file.write("# @Tinteger|integer|integer|string|string|string\n") - out_file.write("# FEATURE_DATA\n") - - with open(path_file_path) as file: - first_line = file.readline().strip().split() - out_file.write(">\n") - out_file.write(f"# @D{first_line[0]}\n") - out_file.write(f"{first_line[4]} {first_line[5]}\n") - for line in file: - # Parse the input fields from the line - fields = line.strip().split() - if len(fields) > 0: - out_file.write(f"{fields[4]} {fields[5]}\n") + try: + proj = osr.SpatialReference() + proj.ImportFromEPSG(int(crs)) + result_wkt = proj.ExportToWkt() + result_wkt = result_wkt.replace('"', '\\"') + result_proj = proj.ExportToProj4() + # print(f'epsg resutl is: {result_wkt}') + # print(f'proj string is: {result_proj}') + del proj + + with open(output_file_path, mode) as out_file: + if mode == 'w': + out_file.write("# @VGMT1.0 @GLINESTRING\n") + out_file.write(f'# @Jp"{result_proj}"\n') + out_file.write(f'# @Jw"{result_wkt}"\n') + out_file.write("# @Nlinenum|flightnum|date|Survey|Company|Status\n") + out_file.write("# @Tinteger|integer|integer|string|string|string\n") + out_file.write("# FEATURE_DATA\n") + + with open(path_file_path) as file: + first_line = file.readline().strip().split() + out_file.write(">\n") + out_file.write(f"# @D{first_line[0]}\n") + out_file.write(f"{first_line[4]} {first_line[5]}\n") + for line in file: + # Parse the input fields from the line + fields = line.strip().split() + if len(fields) > 0: + out_file.write(f"{fields[4]} {fields[5]}\n") + except Exception as e: + print(f"Error processing file {path_file_path}: {e}", file=sys.stderr) + sys.exit(1) def print_boxes(pl, pt, pr, pb, out_file, xpo, ypo): - out_file.write(">\n") - out_file.write("# @Dextent\n") - out_file.write(f"{pl - xpo} {pt + ypo}\n") - out_file.write(f"{pr - xpo} {pt + ypo}\n") - out_file.write(f"{pr - xpo} {pb + ypo}\n") - out_file.write(f"{pl - xpo} {pb + ypo}\n") - out_file.write(f"{pl - xpo} {pt + ypo}\n") - - out_file.write(">\n") - out_file.write("# @Dupper_left\n") - out_file.write(f"{pl - xpo} {pt + ypo}\n") - out_file.write(f"{pl + 1 - xpo} {pt + ypo}\n") - out_file.write(f"{pl + 1 - xpo} {pt - 1 + ypo}\n") - out_file.write(f"{pl - xpo} {pt - 1 + ypo}\n") - out_file.write(f"{pl - xpo} {pt + ypo}\n") - - out_file.write(">\n") - out_file.write("# @Dlower_right\n") - out_file.write(f"{pr - xpo} {pb + ypo}\n") - out_file.write(f"{pr - 1 - xpo} {pb + ypo}\n") - out_file.write(f"{pr - 1 - xpo} {pb + 1 + ypo}\n") - out_file.write(f"{pr - xpo} {pb + 1 + ypo}\n") - out_file.write(f"{pr - xpo} {pb + ypo}\n") - - out_file.write(">\n") - out_file.write("# @Dground_level\n") + try: + out_file.write(">\n") + out_file.write("# @Dextent\n") + out_file.write(f"{pl - xpo} {pt + ypo}\n") + out_file.write(f"{pr - xpo} {pt + ypo}\n") + out_file.write(f"{pr - xpo} {pb + ypo}\n") + out_file.write(f"{pl - xpo} {pb + ypo}\n") + out_file.write(f"{pl - xpo} {pt + ypo}\n") + + out_file.write(">\n") + out_file.write("# @Dupper_left\n") + out_file.write(f"{pl - xpo} {pt + ypo}\n") + out_file.write(f"{pl + 1 - xpo} {pt + ypo}\n") + out_file.write(f"{pl + 1 - xpo} {pt - 1 + ypo}\n") + out_file.write(f"{pl - xpo} {pt - 1 + ypo}\n") + out_file.write(f"{pl - xpo} {pt + ypo}\n") + + out_file.write(">\n") + out_file.write("# @Dlower_right\n") + out_file.write(f"{pr - xpo} {pb + ypo}\n") + out_file.write(f"{pr - 1 - xpo} {pb + ypo}\n") + out_file.write(f"{pr - 1 - xpo} {pb + 1 + ypo}\n") + out_file.write(f"{pr - xpo} {pb + 1 + ypo}\n") + out_file.write(f"{pr - xpo} {pb + ypo}\n") + + out_file.write(">\n") + out_file.write("# @Dground_level\n") + except Exception as e: + print(f"Error writing boxes: {e}", file=sys.stderr) + sys.exit(1) def box_elevation(extent_file_path, path_file_path, output_file_path, depth_lines, line_increments, xpo, ypo): - last = None - yy = [] - - lines = int(depth_lines) - depth = depth_line_increments = int(line_increments) # Set initial value for depth - - # This function will be modified to do the following: - # - Look for all path and extent files in folder - # - check that the numbers are equal, if not report on the ones that are missing. - # - modify jointjs icons to show actual numbers - # - create box.gmt files for all combination of path and extent files. - - with open(output_file_path, 'w') as out_file: - - with open(extent_file_path) as file: - for line in file: - # Parse the input fields from the line - fields = line.strip().split() - if len(fields) > 0: - nr1, pt, pl, pr, pb, nr2, dt, nr3, db = map(float, fields) - y_of = dt - y_fact = (db - dt) / (pb - pt) - - out_file.write("# @VGMT1.0 @GLINESTRING\n") - out_file.write("# @Nlinename\n") - out_file.write("# @Tstring\n") - out_file.write("# FEATURE_DATA\n") - - print_boxes(pl, pt, pr, pb, out_file, xpo, ypo) - - with open(path_file_path) as path_file: - for line in path_file: - path_fields = line.strip().split() - - if len(path_fields) > 0: - ppt = int(path_fields[1]) - py = (y_of - float(path_fields[8])) / y_fact - out_file.write(f"{int(path_fields[1]) - 1} " - f"{round(decimal.Decimal((-py + ypo) - (2 / y_fact)), 4).normalize()}\n") - yy.insert(ppt - 1, py * -1) - last = ppt - - for j in range(1, lines + 1): - out_file.write(">\n") - out_file.write(f"# @D{depth}\n") - - for i in range(0, last): - # ly=(yy[i]-(ddd/y_fact)) - # print i-1" "ly+ypo - - ly = round(decimal.Decimal(str(yy[i] - (depth / y_fact) - (2 / y_fact))), 4).normalize() - out_file.write(f'{i} {round(ly + decimal.Decimal(ypo), 4).normalize()}\n') - - depth += depth_line_increments + try: + last = None + yy = [] + + lines = int(depth_lines) + depth = depth_line_increments = int(line_increments) # Set initial value for depth + + # This function will be modified to do the following: + # - Look for all path and extent files in folder + # - check that the numbers are equal, if not report on the ones that are missing. + # - modify jointjs icons to show actual numbers + # - create box.gmt files for all combination of path and extent files. + + with open(output_file_path, 'w') as out_file: + + with open(extent_file_path) as file: + for line in file: + # Parse the input fields from the line + fields = line.strip().split() + if len(fields) > 0: + nr1, pt, pl, pr, pb, nr2, dt, nr3, db = map(float, fields) + y_of = dt + y_fact = (db - dt) / (pb - pt) + + out_file.write("# @VGMT1.0 @GLINESTRING\n") + out_file.write("# @Nlinename\n") + out_file.write("# @Tstring\n") + out_file.write("# FEATURE_DATA\n") + + print_boxes(pl, pt, pr, pb, out_file, xpo, ypo) + + with open(path_file_path) as path_file: + for line in path_file: + path_fields = line.strip().split() + + if len(path_fields) > 0: + ppt = int(path_fields[1]) + py = (y_of - float(path_fields[8])) / y_fact + out_file.write(f"{int(path_fields[1]) - 1} " + f"{round(decimal.Decimal((-py + ypo) - (2 / y_fact)), 4).normalize()}\n") + yy.insert(ppt - 1, py * -1) + last = ppt + + for j in range(1, lines + 1): + out_file.write(">\n") + out_file.write(f"# @D{depth}\n") + + for i in range(0, last): + # ly=(yy[i]-(ddd/y_fact)) + # print i-1" "ly+ypo + + ly = round(decimal.Decimal(str(yy[i] - (depth / y_fact) - (2 / y_fact))), 4).normalize() + out_file.write(f'{i} {round(ly + decimal.Decimal(ypo), 4).normalize()}\n') + + depth += depth_line_increments + except Exception as e: + print(f"Error processing box elevation: {e}", file=sys.stderr) + sys.exit(1) def main(input_directory, output_directory, crs="28349", gis="esri_arcmap_0.5", lines=10, lines_increment=30): diff --git a/aemworkflow/validation.py b/aemworkflow/validation.py index 700f4c7..d61a340 100644 --- a/aemworkflow/validation.py +++ b/aemworkflow/validation.py @@ -6,10 +6,14 @@ def validation_remove_quotes(bdf_file_path, bdf_out_file_path, logger_session=logger): logger_session.info("Running remove quotes validation.") - with open(bdf_file_path, 'r') as bdf_file, open(bdf_out_file_path, 'w') as bdf_clean_out_file: - for line in bdf_file: - bdf_clean_out_file.write(line.replace('"', '')) - logger_session.info("Completed remove quotes validation.") + try: + with open(bdf_file_path, 'r') as bdf_file, open(bdf_out_file_path, 'w') as bdf_clean_out_file: + for line in bdf_file: + bdf_clean_out_file.write(line.replace('"', '')) + logger_session.info("Completed remove quotes validation.") + except Exception as e: + logger_session.error(f"Error during remove quotes validation: {e}") + raise def validation_qc_units(erc_file_path, bdf_2_file_path, validation_dir, logger_session=logger): @@ -22,94 +26,97 @@ def validation_qc_units(erc_file_path, bdf_2_file_path, validation_dir, logger_s no_unit = {} # Read stratigraphic-unit.csv - qc_outputs_path = os.path.join(validation_dir, 'qc') + os.sep - Path(qc_outputs_path).mkdir(exist_ok=True) - - with open(erc_file_path, "r", encoding='utf-8') as strat_file: - for line in strat_file: - fields = line.strip().split("|") - if len(fields) != 43: - with open(fr"{qc_outputs_path}asud_nf.asc", "a") as nf_file: - nf_file.write(f"{len(fields)} {line}") - else: - stratno[fields[0]] = fields[1] - name[fields[0]] = fields[0] - - # Read AusAEM1_Interp.csv and compare unit name-number - with open(bdf_2_file_path, "r") as interp_file: - with open(fr'{qc_outputs_path}error_list.log', "a") as error_list_file: - - for line in interp_file: - fields = line.strip().split("|") - if len(fields) <= 25: - with open(fr"{qc_outputs_path}short_nf.log", "a") as short_nf_file: - short_nf_file.write(f"{len(fields)} {fields[0]} {fields[1]}\n") + try: + qc_outputs_path = os.path.join(validation_dir, 'qc') + os.sep + Path(qc_outputs_path).mkdir(exist_ok=True) - if fields[7] == '' and fields[8] == '': - if fields[10] == '' and fields[11] == '': - if fields[13] == '' and fields[14] == '': + with open(erc_file_path, "r", encoding='utf-8') as strat_file: + for line in strat_file: + fields = line.strip().split("|") + if len(fields) != 43: + with open(fr"{qc_outputs_path}asud_nf.asc", "a") as nf_file: + nf_file.write(f"{len(fields)} {line}") + else: + stratno[fields[0]] = fields[1] + name[fields[0]] = fields[0] + + # Read AusAEM1_Interp.csv and compare unit name-number + with open(bdf_2_file_path, "r") as interp_file: + with open(fr'{qc_outputs_path}error_list.log', "a") as error_list_file: + + for line in interp_file: + fields = line.strip().split("|") + if len(fields) <= 25: + with open(fr"{qc_outputs_path}short_nf.log", "a") as short_nf_file: + short_nf_file.write(f"{len(fields)} {fields[0]} {fields[1]}\n") + + if fields[7] == '' and fields[8] == '': + if fields[10] == '' and fields[11] == '': + if fields[13] == '' and fields[14] == '': + units[f"{fields[7]} {fields[8]}"] = f"{fields[7]},{fields[8]}" + count[f"{fields[7]} {fields[8]}"] = count.get(f"{fields[7]} {fields[8]}", 0) + 1 + error_list_file.write(f"nulls|{fields[7]}|{fields[8]}|{line}") + continue + + if (name.get(fields[7]) == fields[7] and stratno.get(fields[7]) == fields[8]) or \ + (name.get(fields[10]) == fields[10] and stratno.get(fields[10]) == fields[11]) or \ + (name.get(fields[13]) == fields[13] and stratno.get(fields[13]) == fields[14]): + + if name.get(fields[7]) == fields[7] and stratno.get(fields[7]) == fields[8]: units[f"{fields[7]} {fields[8]}"] = f"{fields[7]},{fields[8]}" count[f"{fields[7]} {fields[8]}"] = count.get(f"{fields[7]} {fields[8]}", 0) + 1 - error_list_file.write(f"nulls|{fields[7]}|{fields[8]}|{line}") - continue - - if (name.get(fields[7]) == fields[7] and stratno.get(fields[7]) == fields[8]) or \ - (name.get(fields[10]) == fields[10] and stratno.get(fields[10]) == fields[11]) or \ - (name.get(fields[13]) == fields[13] and stratno.get(fields[13]) == fields[14]): - - if name.get(fields[7]) == fields[7] and stratno.get(fields[7]) == fields[8]: - units[f"{fields[7]} {fields[8]}"] = f"{fields[7]},{fields[8]}" - count[f"{fields[7]} {fields[8]}"] = count.get(f"{fields[7]} {fields[8]}", 0) + 1 - else: - no_unit[f"{fields[7]} {fields[8]}"] = f"{fields[7]},{fields[8]}" - count[f"{fields[7]} {fields[8]}"] = count.get(f"{fields[7]} {fields[8]}", 0) + 1 - error_list_file.write(f"over|{fields[7]}|{fields[8]}|{line}") - - if name.get(fields[10]) == fields[10] and stratno.get(fields[10]) == fields[11]: - units[f"{fields[10]} {fields[11]}"] = f"{fields[10]},{fields[11]}" - count[f"{fields[10]} {fields[11]}"] = count.get(f"{fields[10]} {fields[11]}", 0) + 1 - else: - no_unit[f"{fields[10]} {fields[11]}"] = f"{fields[10]},{fields[11]}" - count[f"{fields[10]} {fields[11]}"] = count.get(f"{fields[10]} {fields[11]}", 0) + 1 - error_list_file.write(f"under|{fields[10]}|{fields[11]}|{line}") + else: + no_unit[f"{fields[7]} {fields[8]}"] = f"{fields[7]},{fields[8]}" + count[f"{fields[7]} {fields[8]}"] = count.get(f"{fields[7]} {fields[8]}", 0) + 1 + error_list_file.write(f"over|{fields[7]}|{fields[8]}|{line}") + + if name.get(fields[10]) == fields[10] and stratno.get(fields[10]) == fields[11]: + units[f"{fields[10]} {fields[11]}"] = f"{fields[10]},{fields[11]}" + count[f"{fields[10]} {fields[11]}"] = count.get(f"{fields[10]} {fields[11]}", 0) + 1 + else: + no_unit[f"{fields[10]} {fields[11]}"] = f"{fields[10]},{fields[11]}" + count[f"{fields[10]} {fields[11]}"] = count.get(f"{fields[10]} {fields[11]}", 0) + 1 + error_list_file.write(f"under|{fields[10]}|{fields[11]}|{line}") + + if name.get(fields[13]) == fields[13] and stratno.get(fields[13]) == fields[14]: + units[f"{fields[13]} {fields[14]}"] = f"{fields[13]},{fields[14]}" + count[f"{fields[13]} {fields[14]}"] = count.get(f"{fields[13]} {fields[14]}", 0) + 1 + else: + no_unit[f"{fields[13]} {fields[14]}"] = f"{fields[13]},{fields[14]}" + count[f"{fields[13]} {fields[14]}"] = count.get(f"{fields[13]} {fields[14]}", 0) + 1 + error_list_file.write(f"within|{fields[13]}|{fields[14]}|{line}") - if name.get(fields[13]) == fields[13] and stratno.get(fields[13]) == fields[14]: - units[f"{fields[13]} {fields[14]}"] = f"{fields[13]},{fields[14]}" - count[f"{fields[13]} {fields[14]}"] = count.get(f"{fields[13]} {fields[14]}", 0) + 1 else: - no_unit[f"{fields[13]} {fields[14]}"] = f"{fields[13]},{fields[14]}" - count[f"{fields[13]} {fields[14]}"] = count.get(f"{fields[13]} {fields[14]}", 0) + 1 - error_list_file.write(f"within|{fields[13]}|{fields[14]}|{line}") - - else: - # No match at all - no_unit[f'{fields[7]} {fields[8]}'] = f'{fields[7]},{fields[8]}' - no_unit[f'{fields[10]} {fields[11]}'] = f'{fields[10]},{fields[11]}' - no_unit[f'{fields[13]} {fields[14]}'] = f'{fields[13]},{fields[14]}' - - count[f'{fields[7]} {fields[8]}'] = count.get(f'{fields[7]} {fields[8]}', 0) + 1 - count[f'{fields[10]} {fields[11]}'] = count.get(f'{fields[10]} {fields[11]}', 0) + 1 - count[f'{fields[13]} {fields[14]}'] = count.get(f'{fields[13]} {fields[14]}', 0) + 1 - - error_list_file.write(f'over|{fields[7]}|{fields[8]}|{line}') - error_list_file.write(f'under|{fields[10]}|{fields[11]}|{line}') - error_list_file.write(f'within|{fields[13]}|{fields[14]}|{line}') - - d = date.today().strftime("%Y%m%d") - summary_file = fr'{qc_outputs_path}AEM_validation_summary_{d}.txt' - - with open(summary_file, "a") as summary_file: - logger_session.info("result,name,number,count") - summary_file.write('result,name,number,count\n') - for var in units: - logger_session.info(f"matched,{units[var]},{count[var]}") - summary_file.write(f'matched,{units[var]},{count[var]}\n') - - for var in no_unit: - logger_session.info(f"no match,{no_unit[var]},{count[var]}") - summary_file.write(f'no match,{no_unit[var]},{count[var]}\n') - - logger_session.info("completed qc_units validation.") + # No match at all + no_unit[f'{fields[7]} {fields[8]}'] = f'{fields[7]},{fields[8]}' + no_unit[f'{fields[10]} {fields[11]}'] = f'{fields[10]},{fields[11]}' + no_unit[f'{fields[13]} {fields[14]}'] = f'{fields[13]},{fields[14]}' + + count[f'{fields[7]} {fields[8]}'] = count.get(f'{fields[7]} {fields[8]}', 0) + 1 + count[f'{fields[10]} {fields[11]}'] = count.get(f'{fields[10]} {fields[11]}', 0) + 1 + count[f'{fields[13]} {fields[14]}'] = count.get(f'{fields[13]} {fields[14]}', 0) + 1 + + error_list_file.write(f'over|{fields[7]}|{fields[8]}|{line}') + error_list_file.write(f'under|{fields[10]}|{fields[11]}|{line}') + error_list_file.write(f'within|{fields[13]}|{fields[14]}|{line}') + + d = date.today().strftime("%Y%m%d") + summary_file = fr'{qc_outputs_path}AEM_validation_summary_{d}.txt' + + with open(summary_file, "a") as summary_file: + logger_session.info("result,name,number,count") + summary_file.write('result,name,number,count\n') + for var in units: + logger_session.info(f"matched,{units[var]},{count[var]}") + summary_file.write(f'matched,{units[var]},{count[var]}\n') + + for var in no_unit: + logger_session.info(f"no match,{no_unit[var]},{count[var]}") + summary_file.write(f'no match,{no_unit[var]},{count[var]}\n') + + logger_session.info("completed qc_units validation.") + except Exception as e: + logger_session.error(f"Error during qc_units validation: {e}") def main(input_directory, output_directory, asud): diff --git a/dist/aemworkflow-0.0.3-py3-none-any.whl b/dist/aemworkflow-0.0.3-py3-none-any.whl new file mode 100644 index 0000000..5965ace Binary files /dev/null and b/dist/aemworkflow-0.0.3-py3-none-any.whl differ diff --git a/dist/aemworkflow-0.0.3.tar.gz b/dist/aemworkflow-0.0.3.tar.gz new file mode 100644 index 0000000..66d0dff Binary files /dev/null and b/dist/aemworkflow-0.0.3.tar.gz differ diff --git a/docs/source/installation/install_anaconda.rst b/docs/source/installation/install_anaconda.rst index ef7a96e..d51069c 100644 --- a/docs/source/installation/install_anaconda.rst +++ b/docs/source/installation/install_anaconda.rst @@ -91,6 +91,6 @@ Once `Miniforge` is installed and activated we can now create an environment to .. code-block:: bash conda env create -f environment.yml - conda activate aem_workflow-env + conda activate aemworkflow-env diff --git a/docs/source/run_scripts/index.rst b/docs/source/run_scripts/index.rst index d634c59..49f332a 100644 --- a/docs/source/run_scripts/index.rst +++ b/docs/source/run_scripts/index.rst @@ -11,7 +11,7 @@ Pre-interpretation .. code-block:: bash - aemworkflow pre_interpret "{input_directory}" "{output_directory}" + aemworkflow pre-interpret --i "{input_directory}" --o "{output_directory}" **Parameter examples:** @@ -34,7 +34,7 @@ Interpretation .. code-block:: bash - aemworkflow interpret "{input_directory}" "{output_directory}" + aemworkflow interpret --i "{input_directory}" --o "{output_directory}" **Parameter examples:** @@ -54,7 +54,7 @@ Validation .. code-block:: bash - aemworkflow validate "{input_directory}" "{output_directory}" "{asud_filename}" + aemworkflow validate --i "{input_directory}" --o "{output_directory}" --a "{asud_filename}" **Parameter examples:** @@ -71,7 +71,7 @@ Conversion .. code-block:: bash - aemworkflow convert "{input_directory}" "{output_directory}" + aemworkflow convert --i "{input_directory}" --o "{output_directory}" **Parameter examples:** @@ -89,7 +89,7 @@ Export .. code-block:: bash - aemworkflow export "{input_directory}" "{output_directory}" "{boundary_file}" "{split_file}" -mdc -mdch -egs + aemworkflow export --i "{input_directory}" --o "{output_directory}" --b "{boundary_file}" --s "{split_file}" --mdc --mdch --egs **Parameter examples:** diff --git a/pyproject.toml b/pyproject.toml index f5393af..2ffc9e1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "aemworkflow" -version = "0.0.2" +version = "0.0.3" authors = [ { name="Geoscience Australia", email="toolkits@ga.gov.au" }, ]