ASB-30568: Adding read_product Function to load files from s3 to memory by AlexReedy · Pull Request #3561 · astropy/astroquery

AlexReedy · 2026-03-20T18:07:35Z

Adding in ability to read FITS and ASDF data products to memory from s3:// using Observations.read_product() function

codecov · 2026-04-17T15:45:08Z

Codecov Report

❌ Patch coverage is 84.61538% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.24%. Comparing base (e3702c4) to head (3508915).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
astroquery/mast/observations.py	84.61%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3561      +/-   ##
==========================================
+ Coverage   73.22%   73.24%   +0.01%     
==========================================
  Files         226      226              
  Lines       21000    21026      +26     
==========================================
+ Hits        15378    15400      +22     
- Misses       5622     5626       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bsipocz

The commit history is a bit all over the place. Could you please clean it up and squash to logical chucks rather than quasi random back and forth? Thanks!

I suppose this will also need narrative documentation; but I'll leave the more detailed review to @snbianco

bsipocz · 2026-04-20T19:13:42Z

+                except Exception as e:
+                    log.exception(f"Failed to open ASD File: {product_path} {e}")
+        else:
+            print("Unsupported extension type")


please don't print anything; either raise proper warning or error classes or put this behind a verbose option.

ah thank you! I thought I got rid of all of those

bsipocz · 2026-04-20T19:15:03Z

    '`~astroquery.mast.ObservationsClass.enable_cloud_dataset` method.'
 )

+asdf_packages = ["asdf", "s3fs", "fsspec", "lz4", "gwcs"]


does all of these really need to be checked here even as they are not directly been used? If asdf requires the whole list then these checks should be dealt with upstream in asdf itself.

This has been an ongoing question with the implementation. Required would be asdf, s3fs, fsspec for the function to work. for asdf specifically, lz4 seems to be the primary compression algorithm being used for asdf and gwcs is for the general.

I know that gwcs is in their test environment, and they also make calls to lz4 but I don't believe it's included when it's installed. I agree it should probably be upstream in asdf.

…fits data products from s3

snbianco · 2026-04-29T00:53:07Z

Thanks again for this PR, Alex! Can you add a quick section about this function to the Observations docs (docs/mast/mast_obsquery.rst). You'll probably want to put it in the cloud data access section.

This PR will also need some tests. The non-remote-access tests may be slightly tricky with mocking. Let me look into it and see if I can point you in the right direction. The remote-access tests should be more straightforward.

…required for testing

…on logs and tests

…bove 2.0

…re file extension is no supported

snbianco

Another round of review! This is looking good. A lot of my comments were questions or notes on the docs/docstrings. Let me know if you have questions or want to chat more about the switch to fsspec!

snbianco · 2026-05-11T14:08:58Z

    assert Observations._cloud_enabled_explicitly is False


+@pytest.fixture


The convention we have for fixtures in this file is to put them near the top, before any of the tests. For this one and the s3_asdf_path specifically, though, I'm not sure if we need a fixture, since they're only being used once. I'd rather see the value of s3_fits_path be parametrized.

snbianco · 2026-05-11T14:09:11Z

+
+
+def test_read_product_fits(s3_fits_path, mock_fits_open, mocker):
+    mocker.patch("astropy.__version__", "5.0.0")


Why do we need this line?

snbianco · 2026-05-11T14:13:26Z

+    def test_observations_read_product_asdf(self):
+        asdf = pytest.importorskip("asdf")
+
+        product_path = "s3://stpubdata/roman/nexus/soc_simulations/tutorial_data" \


Are there any smaller files that we have available to us from the Nexus? This takes around 15 seconds to load in, and we try to keep this test suite as short in duration as we can.

snbianco · 2026-05-11T14:15:04Z

+        Parameters
+        ----------
+        product_path: str
+            URI to the product in open bucket.


Suggested change

URI to the product in open bucket.

URI to the product in the STScI S3 open data bucket.

snbianco · 2026-05-11T14:16:07Z

+        product_path: str
+            URI to the product in open bucket.
+        read_as: str, optional
+            How to read the file. Currently only .fits and .asdf is supported by "auto". Defaults to "auto".


Suggested change

How to read the file. Currently only .fits and .asdf is supported by "auto". Defaults to "auto".

How to read the file. Currently only FITS and ASDF file types are supported by "auto". Default is "auto".

snbianco · 2026-05-11T14:30:05Z

+Streaming Data Products from S3 to memory
+-----------------------------------------
+If instead of downloading you would like to load an S3 URI directly to memory you can use `~astroquery.mast.ObservationsClass.read_product`. 
+This function supports FITS and ASDF data products and will automatically parse the file for the suffix and load it to memory using `~astropy.io.fits.open` or ``~asdf.open``.


Suggested change

This function supports FITS and ASDF data products and will automatically parse the file for the suffix and load it to memory using `~astropy.io.fits.open` or ``~asdf.open``.

This function supports FITS and ASDF data products and will automatically parse the file for the suffix and load it to memory using `~astropy.io.fits.open` or `~asdf.open`.

snbianco · 2026-05-11T14:30:30Z

+
+Streaming Data Products from S3 to memory
+-----------------------------------------
+If instead of downloading you would like to load an S3 URI directly to memory you can use `~astroquery.mast.ObservationsClass.read_product`. 


Suggested change

If instead of downloading you would like to load an S3 URI directly to memory you can use `~astroquery.mast.ObservationsClass.read_product`.

If instead of downloading you would like to load an S3 URI directly to memory, you can use the `~astroquery.mast.ObservationsClass.read_product` method.

snbianco · 2026-05-11T14:31:48Z

+-----------------------------------------
+If instead of downloading you would like to load an S3 URI directly to memory you can use `~astroquery.mast.ObservationsClass.read_product`. 
+This function supports FITS and ASDF data products and will automatically parse the file for the suffix and load it to memory using `~astropy.io.fits.open` or ``~asdf.open``.
+For ASDF data products additional packages may be required (e.g lz4 and roman-datamodels for ROMAN data). 


Suggested change

For ASDF data products additional packages may be required (e.g lz4 and roman-datamodels for ROMAN data).

For ASDF data products, additional packages may be required (e.g `~lz4` and `~roman-datamodels` for data from the Roman Space Telescope).

snbianco · 2026-05-11T14:32:41Z

 - The cloud dataset in ``Observations`` is now enabled by default if the ``boto3`` and ``botocore`` packages are installed. This
  default can be overridden by setting the ``enable_cloud_dataset`` configuration option to False. [#3534]

+- Adding in ability to read FITS and ASDF dataproducts to memory from s3:// using ``Observations.read_product()`` function. [#3561]


Suggested change

- Adding in ability to read FITS and ASDF dataproducts to memory from s3:// using ``Observations.read_product()`` function. [#3561]

- Adding in ability to read FITS and ASDF data products to memory from STScI's S3 open data bucket using ``Observations.read_product()`` function. [#3561]

snbianco · 2026-05-11T14:34:06Z

   # Temp workaround for https://github.com/RKrahl/pytest-dependency/issues/91
   "pytest-dependency; platform_system != 'Windows'",
   "pytest-rerunfailures",
   "fsspec[http]",


Suggested change

"fsspec[http,s3]",

bsipocz added the mast label Apr 4, 2026

snbianco self-requested a review April 9, 2026 14:53

AlexReedy marked this pull request as ready for review April 17, 2026 16:57

bsipocz reviewed Apr 20, 2026

View reviewed changes

Adding read_product function to mast.observations to stream asdf and …

eb5a1b2

…fits data products from s3

AlexReedy force-pushed the ASB-30568_read-product-function branch from 12ab349 to eb5a1b2 Compare April 28, 2026 17:08

Alex Reedy added 13 commits May 1, 2026 13:19

Removing unecessary logic for checking packages

b04daad

Adding mock tests for read_product

e4a1991

adding asdf to optional project.optional-dependencies

722dde6

Cleaning various codestlye issues

a19308a

Adding in tests remote data testing for read_product function

ca6e1c2

Adding in lz4, gwcs, roman-datamodel to optional-depends as they are …

7229c04

…required for testing

Adding in section to Observations docs for review, clean up of functi…

5f8e973

…on logs and tests

Updating docs for read_product

6990708

Merge branch 'main' into ASB-30568_read-product-function

6f138b3

tox.ini pinning roman-datamodels and gwcs to prevent numpy versions a…

521d468

…bove 2.0

Updating read_product to parse compressed fits files

c3681a5

Cleaning mast_obsquery blank line causing docs build fail

afb86e2

updating tests to handle cases were fits/asdf cannot be open, and whe…

3508915

…re file extension is no supported

snbianco reviewed May 11, 2026

View reviewed changes

		assert Observations._cloud_enabled_explicitly is False


		@pytest.fixture



		def test_read_product_fits(s3_fits_path, mock_fits_open, mocker):
		mocker.patch("astropy.__version__", "5.0.0")

	URI to the product in open bucket.
	URI to the product in the STScI S3 open data bucket.

	How to read the file. Currently only .fits and .asdf is supported by "auto". Defaults to "auto".
	How to read the file. Currently only FITS and ASDF file types are supported by "auto". Default is "auto".

	This function supports FITS and ASDF data products and will automatically parse the file for the suffix and load it to memory using `~astropy.io.fits.open` or ``~asdf.open``.
	This function supports FITS and ASDF data products and will automatically parse the file for the suffix and load it to memory using `~astropy.io.fits.open` or `~asdf.open`.

	If instead of downloading you would like to load an S3 URI directly to memory you can use `~astroquery.mast.ObservationsClass.read_product`.
	If instead of downloading you would like to load an S3 URI directly to memory, you can use the `~astroquery.mast.ObservationsClass.read_product` method.

	For ASDF data products additional packages may be required (e.g lz4 and roman-datamodels for ROMAN data).
	For ASDF data products, additional packages may be required (e.g `~lz4` and `~roman-datamodels` for data from the Roman Space Telescope).

	- Adding in ability to read FITS and ASDF dataproducts to memory from s3:// using ``Observations.read_product()`` function. [#3561]
	- Adding in ability to read FITS and ASDF data products to memory from STScI's S3 open data bucket using ``Observations.read_product()`` function. [#3561]

Uh oh!

Conversation

AlexReedy commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexReedy Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snbianco commented Apr 29, 2026

Uh oh!

snbianco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexReedy commented Mar 20, 2026 •

edited

Loading

codecov Bot commented Apr 17, 2026 •

edited

Loading

AlexReedy Apr 20, 2026 •

edited

Loading