Skip to content

Download products from event subscription JSON files#3517

Merged
bsipocz merged 3 commits into
astropy:mainfrom
snbianco:subscription-prods
Feb 4, 2026
Merged

Download products from event subscription JSON files#3517
bsipocz merged 3 commits into
astropy:mainfrom
snbianco:subscription-prods

Conversation

@snbianco
Copy link
Copy Markdown
Contributor

@snbianco snbianco commented Feb 2, 2026

Main Objectives:

  • Enhance MastMissions.download_products to accept JSON data from the upcoming MAST subscription service (either as a local JSON file or an in-memory object).
  • Add full MastMissions support for IUE to streamline workflow.

Other Notes:

  • More robust error handling when a user tries to input a filename to MastMissions.download_file when the mission requires a URI. Unfortunately, I don't know of an easy way to tell which missions require this from the API, so the list is hard-coded (for now).
  • MastMissions.download_products looks for dataset or fileset keyword when creating the download path for files. If neither are found, instructs user to use flat=True.
  • Finally got around to making some changes to mast_test.py that simplify mocking and making calls to functions. It looks like a lot of changes, but really, I'm just removing the prepended mast. from everywhere it was used.
  • We recently had a change to one of our services that was evoking a warning in tests that query the PS1 catalog in test_mast_remote.py. I avoid these warnings by selecting certain columns in the tests.

@snbianco snbianco added the mast label Feb 2, 2026
@snbianco snbianco marked this pull request as ready for review February 3, 2026 16:39
@bsipocz bsipocz added this to the 0.4.12 milestone Feb 4, 2026
Copy link
Copy Markdown
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good, thank you!

@bsipocz
Copy link
Copy Markdown
Member

bsipocz commented Feb 4, 2026

devdeps failure is unrelated and has been dealt with separately.

@bsipocz bsipocz merged commit 0a17827 into astropy:main Feb 4, 2026
8 of 9 checks passed

if not isinstance(json_data, (list, tuple)):
raise InvalidQueryError(f'Expected a list of product rows in JSON file at {products}.')
products = Table(rows=json_data)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to deduplicate the json_data since the events coming from the subscriptions database may have many of the same files.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's done later on line 816

Copy link
Copy Markdown
Contributor

@falkben falkben Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i could be misreading the code, i'm not familiar with this codebase, but does remove_duplicate_products only work on uri's? whereas this json data is filesets and datasets?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question and totally fair to ask! remove_duplicate_products actually works on an Astropy Table of products and deduplicates based on the key it’s given (filename). The JSON is converted into a Table earlier in the code, so as long as the JSON format is what we expect, it should work as intended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants