Skip to content

Commit 99807eb

Browse files
mike-ganglFrank Greguskafrankinspacesureshshsv
authored
Develop into Main (1.12.0) (#114)
* Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <[email protected]> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <[email protected]> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <[email protected]> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <[email protected]> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <[email protected]> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <[email protected]> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: sureshshsv <[email protected]> Co-authored-by: sureshshsv <[email protected]> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: Frank Greguska <[email protected]> Co-authored-by: sureshshsv <[email protected]> Co-authored-by: sureshshsv <[email protected]>
1 parent b9090a6 commit 99807eb

9 files changed

+228
-101
lines changed

‎CHANGELOG.md

+6
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,12 @@ All notable changes to this project will be documented in this file.
33

44
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
55

6+
## 1.12.0
7+
### Fixed
8+
- Added EDL based token downloading, removing CMR tokens [98](https://github.com/podaac/data-subscriber/issues/98),
9+
### Added
10+
- Added ability to download by filename [109](https://github.com/podaac/data-subscriber/issues/109) and additional regression testing
11+
612
## 1.11.0
713
### Fixed
814
- Fixed an issue where token-refresh was expecting a dictionary, not a list of tuples

‎Downloader.md

+17-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ For installation and dependency information, please see the [top-level README](R
66

77
```
88
$> podaac-data-downloader -h
9-
usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-e EXTENSIONS] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT]
9+
usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-e EXTENSIONS] [--granule-name GRANULE] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT]
1010
1111
optional arguments:
1212
-h, --help show this help message and exit
@@ -34,6 +34,8 @@ optional arguments:
3434
--offset OFFSET Flag used to shift timestamp. Units are in hours, e.g. 10 or -10.
3535
-e EXTENSIONS, --extensions EXTENSIONS
3636
The extensions of products to download. Default is [.nc, .h5, .zip, .tar.gz]
37+
-gr GRANULE, --granule-name GRANULE
38+
The name of the granule to download. Only one granule name can be specified. Script will download all files matching similar granule name sans extension.
3739
--process PROCESS_CMD
3840
Processing command to run on each downloaded file (e.g., compression). Can be specified multiple times.
3941
--version Display script version information and exit.
@@ -121,6 +123,20 @@ machine urs.earthdata.nasa.gov
121123

122124
## Advanced Usage
123125

126+
### Download data by filename
127+
128+
If you're aware of a file you want to download, you can use the `-gr` option to download by a filename. The `-c` (COLLECTION) and `-d` (directory) options are still required.
129+
130+
The `-gr` option works by taking the file name, removing the suffix and searching for a CMR entry called the granuleUR. Some examples of this include:
131+
132+
| Collection | Filename | CMR GranuleUR |
133+
| --- | ----------- | ----------- |
134+
| MUR25-JPL-L4-GLOB-v04.2 | 20221206090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.nc | 20221206090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2 |
135+
| JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F | S6A_P4_2__HR_STD__NR_077_039_20221212T181728_20221212T182728_F07.nc | S6A_P4_2__HR_STD__NR_077_039_20221212T181728_20221212T182728_F07 |
136+
137+
Because of this behavior, granules without data suffixes and granules where the the UR does not directly follow this convention may not work as anticipated. We will be adding the ability to download by granuleUR in a future enhancement.
138+
139+
124140
### Download data by cycle
125141

126142
Some PO.DAAC datasets are better suited for cycles based search instead of start and end times. To enabled this, we've added 'cycle' based downloading to the data-downloader. The following example will download data from cycle 42:

‎poetry.lock

+50-63
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[tool.poetry]
22
name = "podaac-data-subscriber"
3-
version = "1.11.0"
3+
version = "1.12.0"
44
description = "PO.DAAC Data Subscriber Command Line Tool"
55
authors = ["PO.DAAC <[email protected]>"]
66
readme = "README.md"

0 commit comments

Comments
 (0)