-
Notifications
You must be signed in to change notification settings - Fork 26
subscriptions in the MAST API #159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
a68bdd0
really rough draft of using json
ttdu 6fb276b
add requirements
ttdu cfac453
rename file in notebook to match
ttdu 5462d6e
bump requirements to dev
ttdu 99158b6
fix pep8
ttdu 977aaf0
first draft
ttdu 2facb43
fix requirements
ttdu cc5b2fb
make title more descriptive
ttdu 5b88dbf
add pandas as a requirement for table reading
ttdu 11454d7
address Sam and Scott comments
ttdu bc92ec3
add email
ttdu 64a37df
point to astroquery search
ttdu 2c439b4
fix typo
ttdu 11c8a54
add note about EAP
ttdu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| [ | ||
| { | ||
| "event_id": "a1175c61-00ae-484b-8d82-f02ccf17a0b3", | ||
| "mission": "JWST", | ||
| "fileset": "jw05893001001_02201_00001", | ||
| "filename": "jw05893001001_02201_00001_mirimage_o001_crf.fits", | ||
| "program": 5893, | ||
| "event_type": "Reprocessed", | ||
| "archived_date": "2025-12-03T19:44:10.707", | ||
| "public_release_date": "2025-04-30T18:50:03.000" | ||
| }, | ||
|
|
||
| { | ||
| "event_id": "17a653f9-c2fe-4d71-a3ca-2590123615da", | ||
| "mission": "JWST", | ||
| "fileset": "jw05105001001_02201_00001", | ||
| "filename": "jw05105001001_02201_00001_mirimage_o001_crf.fits", | ||
| "program": 5105, | ||
| "event_type": "New", | ||
| "archived_date": "2026-03-30T17:53:19.695", | ||
| "public_release_date": "2026-03-30T17:52:27.000" | ||
| }, | ||
|
|
||
| { | ||
| "event_id": "09498e44-08b3-4f2e-849a-263301f942a1", | ||
| "mission": "JWST", | ||
| "fileset": "jw05105012002_02201_00001", | ||
| "filename": "jw05105012002_02201_00001_mirimage_rateints.fits", | ||
| "program": 5105, | ||
| "event_type": "Public", | ||
| "archived_date": "2026-03-30T17:54:12.7510000", | ||
| "public_release_date": "2026-03-30T17:53:13" | ||
| } | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| git+https://github.com/astropy/astroquery.git |
192 changes: 192 additions & 0 deletions
192
notebooks/multi_mission/subscriptions/subscriptions.ipynb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "f2d85512-c5cc-4b65-9fa8-f6137d422d8d", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Subscriptions\n", | ||
| "---\n", | ||
| "This is a short notebook, focused on processing the results of a MAST subscription email notification. By the end of this notebook, you will be able to you:\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "- Use astroquery to download products from a subscription notification\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "- Categorize subscriptions, based on the contents of the JSON blob.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "41b1de6b-d616-458b-837f-07988c35bb69", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Imports\n", | ||
|
ttdu marked this conversation as resolved.
|
||
| "\n", | ||
| "There are limited packages needed for this notebook:\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "- `astroquery.mast` to download products. For the latest functionality (and to run the two-line download), you'll need to install directly from the repo: `pip install git+https://github.com/astropy/astroquery.git`\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "- (optional) `json` to open the json blob contained in the subscriptions email" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "0ed6df5c-cbf5-4dcc-a345-014e377b3eef", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Quick Run: Two-Line Download\n", | ||
| "\n", | ||
| "If you are simply interested in getting your products as quickly as possible, this can be done in just two lines of code. All you need to do is point `download_products` at the file you received in your email:" | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "a0daaaca-b99f-42a6-b314-7735cd71d03b", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from astroquery.mast import MastMissions\n", | ||
| "MastMissions.download_products(\"JWST.json\")" | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "8057afe8-de90-4299-8748-f1082ba75e76", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "As with a standard astroquery download, the display will show a set of download notifications, along with a status table of all products after completion.\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "\n", | ||
| "That's it! You're done. Your products are downloaded.\n", | ||
| "\n", | ||
|
ttdu marked this conversation as resolved.
|
||
| "## A Little More Detail\n", | ||
| "\n", | ||
| "Although it is easy to download your products, it is useful to know more about how the download proceeds and the contents of the JSON file.\n", | ||
| "\n", | ||
| "### Examining the JSON Contents\n", | ||
| "\n", | ||
| "Let's a look at the information in the JSON itself. We'll need one more import to accomplish this: `json` will read our file into a list of dictionaries." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "c8e8a137-0879-4a4b-b9a4-7b6bde410ffa", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# we need this to properly read in the file\n", | ||
| "import json\n", | ||
| "\n", | ||
| "# open our file and take a look inside\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "with open(\"JWST.json\", \"r\") as infile:\n", | ||
| " subscriptions = json.load(infile)\n", | ||
| "subscriptions" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "e2bf98c0-6ea8-4e31-965f-c240fe88c809", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "As we would expect from our download., there are three files in this list.\n", | ||
| "\n", | ||
| "Although we can see the entire file here, let's briefly simplify this to understand what information is being provided to us." | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "97ff559d-b4d7-4ceb-aadb-60f869b083cb", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# get the keys of the first Python dictionary\n", | ||
| "keys = subscriptions[0].keys()\n", | ||
| "# print out the keys\n", | ||
| "print(len(keys), keys)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "86bb4a38-1b2e-48d2-a754-330bc476f7dc", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Of these eight keys, it is particularly important to understand the `event_type`, which represents the reason for the subscription notification:\n", | ||
| "- `Reprocessed`: the file was reprocessed by an updated version of a calibration pipeline\n", | ||
| "- `Public`: the file was previously under an [exclusive access period (EAP)](https://archive.stsci.edu/publishing/data-use#h2-24a4fb60-93bc-4b19-803e-e8e7b04e8823), but is now publicly available\n", | ||
| "- `New`: this file is new to the archive\n", | ||
| "\n", | ||
| "In the case where an investigator waives their EAP, files will be considered both new **and** public; notifications for either category will trigger.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "d544622c-151a-4883-8137-520de8fe7094", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "for s in subscriptions:\n", | ||
| " print(s['event_type'])" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "5f9de543-4862-4bf6-b9d8-da539f1bd550", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### A Note on the Download Process\n", | ||
| "\n", | ||
| "Normally, when using the `MissionMast` module, you need to specify the mission to complete a download. For example, if we try to download the first file in our set (`jwst.download_file(\"jw05893001001_02201_00001_mirimage_o001_crf.fits`):" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "88042e7f-3688-41a7-912b-6195a00eef68", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# specify mission first\n", | ||
| "jwst = MastMissions(mission='jwst')\n", | ||
| "# now download file\n", | ||
| "jwst.download_file(\"jw05893001001_02201_00001_mirimage_o001_crf.fits\")" | ||
|
ttdu marked this conversation as resolved.
|
||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "7b897e5f-2fe6-49b1-b411-64a1fbe81179", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Since one of the JSON keys is `mission`, MAST has designed the `download_products` function to read from this directly; no need to add this value manually.\n", | ||
|
ttdu marked this conversation as resolved.
|
||
| "\n", | ||
| "## About this Notebook\n", | ||
| "Issues with this notebook or with your subscription data? Contact [archive@stsci.edu](mailto:archive@stsci.edu) or use the [MAST help desk portal](https://stsci.service-now.com/mast).\n", | ||
| "\n", | ||
| "**Author:** Thomas Dutkiewicz <br>\n", | ||
| "**Keywords:** subscriptions <br>\n", | ||
|
ttdu marked this conversation as resolved.
Outdated
|
||
| "\n", | ||
| "***\n", | ||
| "[Top of Page](#top)\n", | ||
| "<img style=\"float: right;\" src=\"https://raw.githubusercontent.com/spacetelescope/style-guides/master/guides/images/stsci-logo.png\" alt=\"Space Telescope Logo\" width=\"200px\"/> " | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.13.5" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.