Skip to content

Commit 464b469

Browse files
vutrannasavutrannasa
authored
Merge pull request #152 from nasa/DAS-2290-Harmony-update-tests
DAS-2290 Add SPL3SMP_E, SPL3SMP, and SPL3FTP_E tests to HOSS_Regression.ipynb
2 parents 4b19c49 + 10c34da commit 464b469

9 files changed

+283
-13
lines changed

‎CHANGELOG.md

+4
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ versioning. Rather than a static releases, this repository contains of a number
55
of regression tests that are each semi-independent. This CHANGELOG file should be used
66
to document pull requests to this repository.
77

8+
### 2025-03-12 ([#152](https://github.com/nasa/harmony-regression-tests/pull/152))
9+
10+
- Add SPL3SMP_E, SPL3SMP, and SPL3FTP_E tests to HOSS_Regression.ipynb.
11+
812
### 2025-03-06 ([#148](https://github.com/nasa/harmony-regression-tests/pull/148))
913

1014
- Fixes service name key missed in [#147](https://github.com/nasa/harmony-regression-tests/pull/147)

‎test/hoss/HOSS_Regression.ipynb

+266-1
Original file line numberDiff line numberDiff line change
@@ -704,6 +704,271 @@
704704
" print(f'Skipping - HOSS is not configured for this test in {harmony_environment}.')"
705705
]
706706
},
707+
{
708+
"cell_type": "markdown",
709+
"id": "269213d1",
710+
"metadata": {},
711+
"source": [
712+
"### HOSS bounding box spatial subset: <a id=\"spatial-subset\"></a>\n",
713+
"\n",
714+
"This request will limit the spatial extent of the returned output. This request will be fulfilled differently depending on the UMM-S record associated with the data.\n",
715+
"\n",
716+
"SPL3FTP_E is associated with SDS/HOSS Projection-gridded service and will call:\n",
717+
" * `query-cmr` to filter granules to those with matching spatial coverage.\n",
718+
" * `harmony-opendap-subsetter` to perform HOSS operations and extract a rectangular portion of the longitude latitude grid. This will match the bounding box.\n",
719+
" * `maskfill-harmony` to fill any pixels in the rectangular array segment.\n",
720+
"\n",
721+
"TODO: Add Environment.PROD once NSIC updates SPL3FTP_E in the production environment.\n"
722+
]
723+
},
724+
{
725+
"cell_type": "code",
726+
"execution_count": null,
727+
"id": "c3880303",
728+
"metadata": {},
729+
"outputs": [],
730+
"source": [
731+
"hoss_bbox_spatial_non_prod_information = {\n",
732+
" 'collection': Collection(id='C1268616149-EEDTEST'),\n",
733+
" 'granule_id': 'G1268616175-EEDTEST',\n",
734+
"}\n",
735+
"\n",
736+
"hoss_bbox_spatial_prod_information = {\n",
737+
" 'collection': Collection(id='C2776463920-NSIDC_ECS'),\n",
738+
" 'granule_id': 'G2903366136-NSIDC_ECS',\n",
739+
"}\n",
740+
"\n",
741+
"hoss_bbox_spatial_env = {\n",
742+
" Environment.LOCAL: hoss_bbox_spatial_non_prod_information,\n",
743+
" Environment.SIT: hoss_bbox_spatial_non_prod_information,\n",
744+
" Environment.UAT: hoss_bbox_spatial_non_prod_information,\n",
745+
"}\n",
746+
"\n",
747+
"if harmony_environment in hoss_bbox_spatial_env:\n",
748+
" hoss_bbox_spatial_info = hoss_bbox_spatial_env[harmony_environment]\n",
749+
"else:\n",
750+
" hoss_bbox_spatial_info = None"
751+
]
752+
},
753+
{
754+
"cell_type": "code",
755+
"execution_count": null,
756+
"id": "0c3f8a10",
757+
"metadata": {},
758+
"outputs": [],
759+
"source": [
760+
"if test_is_configured(hoss_bbox_spatial_info, 'collection'):\n",
761+
" hoss_bbox_spatial_file_name = 'hoss_bbox_spatial.nc4'\n",
762+
" hoss_bbox_spatial_bbox = BBox(w=-70, s=60, e=-10, n=85)\n",
763+
" hoss_bbox_spatial_request = Request(\n",
764+
" collection=hoss_bbox_spatial_info['collection'],\n",
765+
" granule_id=[hoss_bbox_spatial_info['granule_id']],\n",
766+
" variables=[\n",
767+
" 'Freeze_Thaw_Retrieval_Data_Global/altitude_dem',\n",
768+
" 'Freeze_Thaw_Retrieval_Data_Global/transition_state_flag',\n",
769+
" 'Freeze_Thaw_Retrieval_Data_Polar/altitude_dem',\n",
770+
" 'Freeze_Thaw_Retrieval_Data_Polar/transition_state_flag',\n",
771+
" ],\n",
772+
" spatial=hoss_bbox_spatial_bbox,\n",
773+
" )\n",
774+
"\n",
775+
" submit_and_download(\n",
776+
" harmony_client, hoss_bbox_spatial_request, hoss_bbox_spatial_file_name\n",
777+
" )\n",
778+
" assert exists(\n",
779+
" hoss_bbox_spatial_file_name\n",
780+
" ), 'Unsuccessful HOSS bounding box spatial request.'\n",
781+
"\n",
782+
" compare_results_to_reference_file(\n",
783+
" hoss_bbox_spatial_file_name,\n",
784+
" 'reference_files/hoss_bbox_spatial_reference.nc4',\n",
785+
" '/',\n",
786+
" )\n",
787+
"\n",
788+
" print_success('HOSS bounding box spatial request.')\n",
789+
"else:\n",
790+
" print(f'Skipping - HOSS is not configured for this test in {harmony_environment}.')"
791+
]
792+
},
793+
{
794+
"cell_type": "markdown",
795+
"id": "f25e7fb2",
796+
"metadata": {},
797+
"source": [
798+
"### HOSS polygon spatial subset for 3D variables: <a id=\"spatial-subset\"></a>\n",
799+
"\n",
800+
"A polygon spatial subset for 3D variables:\n",
801+
" This request for a polygon subset of 3D variables in a SPL3SMP granule will be fulfilled using three steps. This can be observed via https://harmony.uat.earthdata.nasa.gov/workflow-ui endpoint of Harmony.\n",
802+
"\n",
803+
"* `query-cmr` to filter granules to those with matching spatial coverage.\n",
804+
"* `harmony-opendap-subsetter` to perform HOSS operations and extract a rectangular portion of the longitude latitude grid. This will match the bounding box.\n",
805+
"* `sds/maskfill-harmony` to fill any pixels in the rectangular array segment, but outside the GeoJSON shape.\n",
806+
"\n",
807+
"TODO: Add Environment.PROD once NSIC updates SPL3SMP in the production environment.\n"
808+
]
809+
},
810+
{
811+
"cell_type": "code",
812+
"execution_count": null,
813+
"id": "0aacd181",
814+
"metadata": {},
815+
"outputs": [],
816+
"source": [
817+
"hoss_shape_file_polygon_spatial_3d_non_prod_information = {\n",
818+
" 'collection': Collection(id='C1268452378-EEDTEST'),\n",
819+
" 'granule_id': 'G1268452388-EEDTEST',\n",
820+
" 'shape_file_path': 'usa-mainland.geojson',\n",
821+
"}\n",
822+
"\n",
823+
"hoss_shape_file_polygon_spatial_3d_prod_information = {\n",
824+
" 'collection': Collection(id='C2776463935-NSIDC_ECS'),\n",
825+
" 'granule_id': 'G2903715759-NSIDC_ECS',\n",
826+
"}\n",
827+
"\n",
828+
"hoss_shape_file_polygon_spatial_3d_env = {\n",
829+
" Environment.LOCAL: hoss_shape_file_polygon_spatial_3d_non_prod_information,\n",
830+
" Environment.SIT: hoss_shape_file_polygon_spatial_3d_non_prod_information,\n",
831+
" Environment.UAT: hoss_shape_file_polygon_spatial_3d_non_prod_information,\n",
832+
"}\n",
833+
"\n",
834+
"if harmony_environment in hoss_shape_file_polygon_spatial_3d_env:\n",
835+
" hoss_shape_file_polygon_spatial_3d_info = hoss_shape_file_polygon_spatial_3d_env[\n",
836+
" harmony_environment\n",
837+
" ]\n",
838+
"else:\n",
839+
" hoss_shape_file_polygon_spatial_3d_info = None"
840+
]
841+
},
842+
{
843+
"cell_type": "code",
844+
"execution_count": null,
845+
"id": "dfddef43",
846+
"metadata": {},
847+
"outputs": [],
848+
"source": [
849+
"if test_is_configured(hoss_shape_file_polygon_spatial_3d_info, 'collection'):\n",
850+
" hoss_shape_file_polygon_spatial_3d_file_name = 'hoss_shape_file_subset_3d.nc4'\n",
851+
" hoss_shape_file_polygon_spatial_3d_request = Request(\n",
852+
" collection=hoss_shape_file_polygon_spatial_3d_info['collection'],\n",
853+
" granule_id=hoss_shape_file_polygon_spatial_3d_info['granule_id'],\n",
854+
" shape=hoss_shape_file_polygon_spatial_3d_info['shape_file_path'],\n",
855+
" variables=[\n",
856+
" 'Soil_Moisture_Retrieval_Data_AM/landcover_class',\n",
857+
" 'Soil_Moisture_Retrieval_Data_PM/landcover_class_pm',\n",
858+
" ],\n",
859+
" )\n",
860+
"\n",
861+
" submit_and_download(\n",
862+
" harmony_client,\n",
863+
" hoss_shape_file_polygon_spatial_3d_request,\n",
864+
" hoss_shape_file_polygon_spatial_3d_file_name,\n",
865+
" )\n",
866+
"\n",
867+
" assert exists(\n",
868+
" hoss_shape_file_polygon_spatial_3d_file_name\n",
869+
" ), 'Unsuccessful HOSS polygon spatial subset for 3D variables request.'\n",
870+
"\n",
871+
" compare_results_to_reference_file(\n",
872+
" hoss_shape_file_polygon_spatial_3d_file_name,\n",
873+
" 'reference_files/hoss_shape_file_subset_3d_reference.nc4',\n",
874+
" '/',\n",
875+
" )\n",
876+
"\n",
877+
" print_success('HOSS polygon spatial subset for 3D variables request.')\n",
878+
"else:\n",
879+
" print(f'Skipping - HOSS is not configured for this test in {harmony_environment}.')"
880+
]
881+
},
882+
{
883+
"cell_type": "markdown",
884+
"id": "2a33fb2e",
885+
"metadata": {},
886+
"source": [
887+
"### HOSS polygon spatial subset: <a id=\"spatial-subset\"></a>\n",
888+
"\n",
889+
"This request will limit the spatial extent of the returned output. This request for a polygon subset in a SPL3SMP_E granule using three steps. This can be observed via https://harmony.uat.earthdata.nasa.gov/workflow-ui endpoint of Harmony.\n",
890+
"\n",
891+
"* `query-cmr` to filter granules to those with matching spatial coverage.\n",
892+
"* `harmony-opendap-subsetter` to perform HOSS operations and extract a rectangular portion of the longitude latitude grid. This will match the bounding box.\n",
893+
"* `sds/maskfill-harmony` to fill any pixels in the rectangular array segment, but outside the GeoJSON shape.\n",
894+
"\n",
895+
"TODO: Add Environment.PROD once NSIC updates SPL3SMP_E in the production environment.\n"
896+
]
897+
},
898+
{
899+
"cell_type": "code",
900+
"execution_count": null,
901+
"id": "fe62d797",
902+
"metadata": {},
903+
"outputs": [],
904+
"source": [
905+
"hoss_shape_file_polygon_spatial_non_prod_information = {\n",
906+
" 'collection': Collection(id='C1268399578-EEDTEST'),\n",
907+
" 'granule_id': 'G1268399648-EEDTEST',\n",
908+
" 'shape_file_path': 'usa-mainland.geojson',\n",
909+
"}\n",
910+
"\n",
911+
"hoss_shape_file_polygon_spatial_prod_information = {\n",
912+
" 'collection': Collection(id='C2776463943-NSIDC_ECS'),\n",
913+
" 'granule_id': 'G2906217596-NSIDC_ECS',\n",
914+
"}\n",
915+
"\n",
916+
"hoss_shape_file_polygon_spatial_env = {\n",
917+
" Environment.LOCAL: hoss_shape_file_polygon_spatial_non_prod_information,\n",
918+
" Environment.SIT: hoss_shape_file_polygon_spatial_non_prod_information,\n",
919+
" Environment.UAT: hoss_shape_file_polygon_spatial_non_prod_information,\n",
920+
"}\n",
921+
"\n",
922+
"if harmony_environment in hoss_shape_file_polygon_spatial_env:\n",
923+
" hoss_shape_file_polygon_spatial_info = hoss_shape_file_polygon_spatial_env[\n",
924+
" harmony_environment\n",
925+
" ]\n",
926+
"else:\n",
927+
" hoss_shape_file_polygon_spatial_info = None"
928+
]
929+
},
930+
{
931+
"cell_type": "code",
932+
"execution_count": null,
933+
"id": "bf563824",
934+
"metadata": {},
935+
"outputs": [],
936+
"source": [
937+
"if test_is_configured(hoss_shape_file_polygon_spatial_info, 'collection'):\n",
938+
" hoss_shape_file_polygon_spatial_file_name = 'hoss_shape_file_subset.nc4'\n",
939+
" hoss_shape_file_polygon_spatial_request = Request(\n",
940+
" collection=hoss_shape_file_polygon_spatial_info['collection'],\n",
941+
" granule_id=hoss_shape_file_polygon_spatial_info['granule_id'],\n",
942+
" shape=hoss_shape_file_polygon_spatial_info['shape_file_path'],\n",
943+
" variables=[\n",
944+
" 'Soil_Moisture_Retrieval_Data_AM/surface_flag',\n",
945+
" 'Soil_Moisture_Retrieval_Data_PM/surface_flag_pm',\n",
946+
" 'Soil_Moisture_Retrieval_Data_Polar_AM/surface_flag',\n",
947+
" 'Soil_Moisture_Retrieval_Data_Polar_PM/surface_flag_pm',\n",
948+
" ],\n",
949+
" )\n",
950+
"\n",
951+
" submit_and_download(\n",
952+
" harmony_client,\n",
953+
" hoss_shape_file_polygon_spatial_request,\n",
954+
" hoss_shape_file_polygon_spatial_file_name,\n",
955+
" )\n",
956+
"\n",
957+
" assert exists(\n",
958+
" hoss_shape_file_polygon_spatial_file_name\n",
959+
" ), 'Unsuccessful HOSS polygon spatial subset request.'\n",
960+
"\n",
961+
" compare_results_to_reference_file(\n",
962+
" hoss_shape_file_polygon_spatial_file_name,\n",
963+
" 'reference_files/hoss_shape_file_subset_reference.nc4',\n",
964+
" '/',\n",
965+
" )\n",
966+
"\n",
967+
" print_success('HOSS polygon spatial subset request.')\n",
968+
"else:\n",
969+
" print(f'Skipping - HOSS is not configured for this test in {harmony_environment}.')"
970+
]
971+
},
707972
{
708973
"cell_type": "markdown",
709974
"id": "ca88eb71-2526-4b34-8ff7-441f869244e0",
@@ -740,7 +1005,7 @@
7401005
"name": "python",
7411006
"nbconvert_exporter": "python",
7421007
"pygments_lexer": "ipython3",
743-
"version": "3.9.13"
1008+
"version": "3.10.16"
7441009
},
7451010
"name": "SDS_Regression.ipynb"
7461011
},

‎test/hoss/environment.yaml

+8-8
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@ channels:
33
- conda-forge
44
- nodefaults
55
dependencies:
6-
- python=3.10
7-
- notebook=6.5.4
8-
- papermill=2.3.4
9-
- python-dateutil=2.7.5
10-
- numpy=1.24.3
11-
- netCDF4=1.6.3
12-
- xarray=2022.12.0
6+
- python=3.12
7+
- notebook=7.2.2
8+
- papermill=2.6.0
9+
- python-dateutil=2.8.2
10+
- numpy=2.1.3
11+
- netCDF4=1.7.2
1312
- pip:
14-
- harmony-py
13+
- harmony-py==1.0.0
14+
- xarray==2025.01.2

‎test/hoss/reference_files/hoss_bbox_spatial_reference.nc4

1.48 MB
Binary file not shown.

‎test/hoss/reference_files/hoss_shape_file_subset_3d_reference.nc4

46.8 KB
Binary file not shown.

‎test/hoss/reference_files/hoss_shape_file_subset_reference.nc4

1.33 MB
Binary file not shown.

‎test/hoss/usa-mainland.geojson

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"type":"FeatureCollection","features":[{"type":"Feature","properties":{},"geometry":{"coordinates":[[[-68.32973979215876,47.30723195123008],[-71.36958513432359,45.27962383795759],[-75.64393375906197,44.66265348989444],[-76.92055043776047,43.40816638889626],[-82.76309080391172,41.66218422548954],[-82.30909659091134,43.58054609339402],[-82.60377996791576,45.403566972205965],[-88.54163358187128,48.26280433166039],[-91.35820659316454,48.256832531967405],[-95.31616991948158,49.24113746490241],[-105.80593785881507,48.96219402625451],[-113.09720446803735,49.05793751602215],[-123.13652588442625,49.18354587731778],[-124.8325853267317,48.28931806653935],[-123.57810514390849,46.10490521001904],[-123.91842051049056,41.39799465302312],[-123.84051962971105,40.116475874883236],[-123.24055987351142,38.73665088439125],[-120.37063327290804,34.93164813680947],[-120.15589207199622,34.17892295794478],[-116.81643979235551,32.66827689269209],[-114.50408927973717,32.76983523925392],[-110.99925851716208,31.24682888940626],[-106.46172782380825,31.63573156310609],[-103.81077941089023,29.47059725218439],[-103.20740839763437,28.93014528727774],[-102.16895502142881,30.121575471718046],[-100.63233173156436,29.41186933687655],[-99.96970381521913,28.344434782586163],[-98.6816162957744,26.430301364861208],[-97.21262763907419,25.907608000603616],[-97.41427762556339,27.656114609949256],[-95.2624790963352,28.959439489795983],[-94.5976455796733,29.637148705721074],[-91.69456043497019,29.68482216143788],[-89.96973076203835,29.176718179955344],[-89.05538688119532,30.50330909133038],[-85.13712632828567,30.27917156271654],[-84.85524599388484,29.403759880934388],[-83.78021512253662,29.97082488168128],[-82.65076267585538,28.434943755761097],[-82.72210226254778,27.854223033078966],[-80.76642092001231,25.170966133742397],[-79.67725703578188,25.022778335591255],[-81.62635749402101,30.626416659029104],[-80.93652325634376,32.272008098873826],[-76.28864026332579,34.82215826419754],[-75.91581262232707,36.66829945757297],[-74.83325081322496,39.20425800602467],[-73.86827967125876,41.059093731552935],[-69.97450841050289,41.40036210390781],[-70.75829791279604,43.21144101275243],[-67.04626711635257,44.603097529704456],[-68.32973979215876,47.30723195123008]]],"type":"Polygon"}}]}

‎test/hoss/utilities.py

+3-3
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
from typing import Dict, Optional
99

1010
from harmony import Client, Request
11-
from harmony.harmony import ProcessingFailedException
11+
from harmony.client import ProcessingFailedException
1212
import xarray as xr
1313

1414

@@ -22,8 +22,8 @@ def compare_results_to_reference_file(
2222
groups, see same-named function in `variable-subsetter/utilities.py`.
2323
2424
"""
25-
reference_data = xr.open_dataset(reference_file_name, group=group_name)
26-
results_data = xr.open_dataset(results_file_name, group=group_name)
25+
reference_data = xr.open_datatree(reference_file_name, group=group_name)
26+
results_data = xr.open_datatree(results_file_name, group=group_name)
2727

2828
assert results_data.equals(reference_data), (
2929
'Output and reference files ' 'do not match.'

‎test/hoss/version.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.1.9
1+
0.1.10

0 commit comments

Comments
 (0)