diff --git a/NetCDF File Fixer.ipynb b/NetCDF File Fixer.ipynb new file mode 100644 index 0000000..d6465dc --- /dev/null +++ b/NetCDF File Fixer.ipynb @@ -0,0 +1,203 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# NetCDF File Reformat for ArcGIS Pro Tutorial Notebook" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Summary\n", + "The purpose of this notebooks is to show users of suborbital campaign data in NetCDF format how to reformat input files to make them useable in ArcGIS Pro as a Multidimensional Dataset. Previously, users have given feedback that NetCDF files do not plot properly as Multidimensional Datasets in ArcGIS Pro - this notebook highlights the root cause and demonstrates how to restructure the file to work seemlessly in ArcGIS Pro. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "**Please create a \"nc_files\" folder to house input NetCDF files, and create a \"new_nc_files\" folder to house converted NetCDF files.**\n", + "- netCDF4\n", + "- numpy\n", + "- pathlib\n", + "- datetime\n", + "- tqdm\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Notebook Author/Affiliation\n", + "Gabriel Mojica/Atmospheric Science Data Center (ASDC)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Steps\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Import required packages

\n", + "For packages you don't have, run a pip install to add them to your machine" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "from netCDF4 import Dataset\n", + "import numpy as np\n", + "from pathlib import Path\n", + "from datetime import datetime, time\n", + "import tqdm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Define directories to iterate through and write to. Ensure you have the source and output directory folders created" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "source_dir = 'nc_files/'\n", + "output_dir = 'new_nc_files/'\n", + "input_path = Path(source_dir)\n", + "input_list = list(input_path.iterdir())\n", + "output_path = Path(output_dir)\n", + "output_list = list(output_path.iterdir())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Build function to copy dimensions, attributes, and variables from src_group to dst_group

\n", + "Copy group attributes with the setncatts() function

\n", + "Copy dimensions with the createDimension() function

\n", + "Copy variables with condition to change \"flag\" variables from float64 to int32 datatypes

\n", + "- Check if variable name contains \"flag\" and is float64, change to int32 then fix NaN values\n", + "- Create the variables in the newly built group\n", + "- Copy variable attributes to the new variables\n", + "- Write in the data\n", + "

Create the subgroups with the createGroup() function

\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "def copy_nc_structure(src_group, dst_group):\n", + " \n", + " dst_group.setncatts(src_group.__dict__)\n", + "\n", + " for name, dimension in src_group.dimensions.items():\n", + " dst_group.createDimension(\n", + " name, (len(dimension) if not dimension.isunlimited() else None)\n", + " )\n", + "\n", + " for name, var in src_group.variables.items():\n", + " if 'flag' in name.lower() and var.datatype == np.float64:\n", + " datatype = np.int32\n", + " data = np.nan_to_num(var[:], nan=-1).astype(np.int32)\n", + " else:\n", + " datatype = var.datatype\n", + " data = var[:]\n", + "\n", + " new_var = dst_group.createVariable(name, datatype, var.dimensions, zlib=True)\n", + "\n", + " new_var.setncatts(var.__dict__)\n", + "\n", + " new_var[:] = data\n", + "\n", + " for group_name, sub_group in src_group.groups.items():\n", + " new_sub_group = dst_group.createGroup(group_name)\n", + " copy_nc_structure(sub_group, new_sub_group)\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once you create the function, you can now run it using an if loop" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if len(input_list) >0:\n", + " print(f\"Processing {len(input_list)} NetCDF Files.\")\n", + " for file in tqdm.tqdm(input_list):\n", + " with Dataset(str(input_path) + '/' + file.name, 'r') as src, Dataset(str(output_path) + '/' + file.name, 'w') as dst:\n", + " copy_nc_structure(src, dst)\n", + " print(f\"Your cleaned NetCDF files have been created in {output_path}.\")\n", + "else:\n", + " print(f\"There are no NetCDF files in {input_list}.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that you've created the new NetCDF files, you can delete the files in your nc_files directory by running the following cell:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "for item in tqdm.tqdm(input_list):\n", + " if item.is_file():\n", + " item.unlink()\n", + "print(\"Your old NetCDF files have been deleted.\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "ArcGISPro", + "language": "python", + "name": "python3" + }, + "language_info": { + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "version": "3.13.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}