Download products from event subscription JSON files#3517
Conversation
bsipocz
left a comment
There was a problem hiding this comment.
This all looks good, thank you!
|
devdeps failure is unrelated and has been dealt with separately. |
|
|
||
| if not isinstance(json_data, (list, tuple)): | ||
| raise InvalidQueryError(f'Expected a list of product rows in JSON file at {products}.') | ||
| products = Table(rows=json_data) |
There was a problem hiding this comment.
we need to deduplicate the json_data since the events coming from the subscriptions database may have many of the same files.
There was a problem hiding this comment.
That's done later on line 816
There was a problem hiding this comment.
i could be misreading the code, i'm not familiar with this codebase, but does remove_duplicate_products only work on uri's? whereas this json data is filesets and datasets?
There was a problem hiding this comment.
Good question and totally fair to ask! remove_duplicate_products actually works on an Astropy Table of products and deduplicates based on the key it’s given (filename). The JSON is converted into a Table earlier in the code, so as long as the JSON format is what we expect, it should work as intended.
Main Objectives:
MastMissions.download_productsto accept JSON data from the upcoming MAST subscription service (either as a local JSON file or an in-memory object).MastMissionssupport for IUE to streamline workflow.Other Notes:
MastMissions.download_filewhen the mission requires a URI. Unfortunately, I don't know of an easy way to tell which missions require this from the API, so the list is hard-coded (for now).MastMissions.download_productslooks fordatasetorfilesetkeyword when creating the download path for files. If neither are found, instructs user to useflat=True.mast_test.pythat simplify mocking and making calls to functions. It looks like a lot of changes, but really, I'm just removing the prependedmast.from everywhere it was used.test_mast_remote.py. I avoid these warnings by selecting certain columns in the tests.