Access to NOIRLab Astro Data Archive by weaverba137 · Pull Request #3359 · astropy/astroquery

weaverba137 · 2025-06-26T22:12:36Z

This PR closes #1701.

The initial commit simply restores the NOIRLab files to the state and content they were when #1701 was last worked on. Further work and testing will be needed to meet astroquery standards.

weaverba137 · 2025-06-26T22:42:39Z

TO DO items identified (i.e. before even starting the review process):

Proofread noirlab/noirlab.rst and verify examples.
Remove telescope holdings from init file. These are certainly outdated. Link to the about page.
Check required acknowledgment(s).
Get remote tests working.
Add non-remote tests to increase coverage, e.g. monkeypatch.

codecov · 2025-06-26T23:06:48Z

Codecov Report

❌ Patch coverage is 93.75000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.39%. Comparing base (7de7c4e) to head (8457192).
⚠️ Report is 225 commits behind head on main.

Files with missing lines	Patch %	Lines
astroquery/noirlab/core.py	93.25%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3359      +/-   ##
==========================================
+ Coverage   72.29%   72.39%   +0.10%     
==========================================
  Files         217      219       +2     
  Lines       20336    20432      +96     
==========================================
+ Hits        14702    14792      +90     
- Misses       5634     5640       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

weaverba137 · 2025-06-26T23:19:27Z

Note to self: tox -e py310-test-online -- -P noirlab to do remote tests on only astroquery.noirlab.

bsipocz · 2025-06-27T18:38:37Z

You also need -R to run the remote tests

weaverba137 · 2025-07-17T23:56:14Z

@keflavich @bsipocz this is ready for first-pass review.

I'm having some trouble with the doc build:

reading sources... [100%] xmatch/xmatch

/home/docs/checkouts/readthedocs.org/user_builds/astroquery/checkouts/3359/docs/noirlab/noirlab.rst:171: CRITICAL: Title level inconsistent:

astroquery.noirlab Package
-------------------------- [docutils]
/home/docs/checkouts/readthedocs.org/user_builds/astroquery/checkouts/3359/docs/noirlab/noirlab.rst:176: CRITICAL: Title level inconsistent:

Classes
^^^^^^^ [docutils]
looking for now-outdated files... none found

I've tried to set up the title levels similar to other packages, such as sdss.

I'm also looking for advice on whether further application of async_to_sync is needed.

weaverba137 · 2025-07-17T23:58:06Z

PS, the .retrieve() method needs further testing. In the previous PR, this method actually returned a HDUList rather than a filename or file object. Could you please advise on how this should be handled in line with similar astroquery methods?

bsipocz · 2025-07-18T00:07:51Z

Thanks. I need to get back to the ESO review first, but put this on the list now, too.

bsipocz

We did a partial review with some comments, I'm coming back to finish this later this week so no need to address any of these yet.

bsipocz · 2025-08-12T19:50:50Z

+    def query_region_async(self, coordinate, radius=0.1, hdu=False, cache=True):
+        """Query for NOIRLab observations by region of the sky.
+
+        Given a sky coordinate and radius, returns a `~astropy.table.Table`
+        of NOIRLab observations.
+
+        Parameters
+        ----------
+        coordinate : :class:`str` or `~astropy.coordinates` object
+            The target region which to search. It may be specified as a
+            string or as the appropriate `~astropy.coordinates` object.
+        radius : :class:`str` or `~astropy.units.Quantity` object, optional
+            Default 0.1 degrees.
+            The string must be parsable by `~astropy.coordinates.Angle`. The
+            appropriate `~astropy.units.Quantity` object from
+            `~astropy.units` may also be used.
+        hdu : :class:`bool`, optional
+            If ``True`` return the URL for HDU-based queries.
+        cache : :class:`bool`, optional
+            If ``True`` cache the result locally.
+
+        Returns
+        -------
+        :class:`~requests.Response`
+            Response object.
+        """
+        self._validate_version()
+        ra, dec = coordinate.to_string('decimal').split()
+        url = f'{self._sia_url(hdu=hdu)}?POS={ra},{dec}&SIZE={radius}&VERB=3&format=json'
+        response = self._request('GET', url, timeout=self.TIMEOUT, cache=cache)
+        # response.raise_for_status()
+        return response


We may want to have async as a kwarg instead and not have two separate methods. See how we do this in some of the other modules e.g. in cadc or ipac.irsa (or the esa modules).

As far as I can tell cadc still has the old @async_to_sync setup. I took a look at ipac.irsa, and there it is apparent that the meaning of async_job is to submit asynchronous queries to a job system of some kind. That doesn't apply to the NOIRLab API.

Based on my experience with sdss, historically "async" meant "return the response object from self._request(), while "sync" meant "based on the response object, download the files referenced in it". This is definitely more applicable to NOIRLab.

Can you elaborate further on the different meanings of "async"?

Based on my experience with sdss, historically "async" meant "return the response object from self._request(), while "sync" meant "based on the response object, download the files referenced in it". This is definitely more applicable to NOIRLab.

This is correct. For the VO case, it can be different, but your interpretation is what I mean by sync vs async.

We have decided that, moving forward, we should not use the async_to_sync functionality, as it is confusing to developers and isn't widely used (#2598). Instead, we prefer kwarg.

It will be a while before we get around to refactoring the old stuff.

@keflavich is there an example of a module where async_to_sync functionality has been replaced with a keyword?

yes, modules (e.g. esa, irsa, simbad) do use the async_job kwarg and then connect to the actual async endpoint to the server.

We have decided that, moving forward, we should not use the async_to_sync functionality, as it is confusing to developers and isn't widely used

I would also add that it wasn't real async, the jobs were not run async on the VO service side, and the duplicated methods were not that great from the user POV either.

@bsipocz, OK, but that's not the case I'm interested in here. I'm referring to "async" methods that would return the raw response rather than doing further conversion into a table or other high-level data. SDSS has several examples of this. See also @keflavich's comment above.

weaverba137 · 2025-08-13T00:03:21Z

@bsipocz, thank you, the suggestions all look good so far, but given my schedule it will be easier to address them all when you're done.

bsipocz · 2025-08-13T00:11:48Z

Yes, totally understandable. I really just submitted this half baked review as that is how far we got during the review tutorial during the summer school I was teaching at. I'll get back to the rest later and will write up an actually usable summary, as there are a couple of things that will need to be fixed while the rest is really just some potential follow-up topics.

And again, thanks for working on this, getting a working noirlab module will be superb!

bsipocz

I have some more comments to finish up my review.

The overarching theme is to make the API a little bit more consistent with the other modules. And maybe consider using pyvo, but that one is optional for this PR.

And in the meantime it looks like the upstream API has changed, so that may mean some tweaks will be necessary (though the tests are primarily failing with the API number comparison).

Also, the docs build runs into some sphinx errors due to API heading inconsistencies, I'm happy to fix up those before merging.

weaverba137 · 2025-09-13T16:57:54Z

Thank you, I was planning to pick this up next week anyway, so this is good timing.

weaverba137 · 2025-09-17T21:27:11Z

Commit 64b84bf addresses some of the review comments. Others will be addressed soon.

There is an issue that needs to be investigated on the NOIRLab side though. Since I last tested in July, when metadata fields are requested in a query, e.g. md5sum, the field is returned, but a duplicate field with header file:md5sum is also returned. This behavior was unexpected both by me and the developer I mentioned this to.

weaverba137 · 2025-09-17T21:34:47Z

Also, I have no clue why the noirlab.rst is causing a RtD error, because the headers are consistent with other files.

bsipocz · 2025-09-18T03:27:58Z

Also, I have no clue why the noirlab.rst is causing a RtD error, because the headers are consistent with other files.

That error rings a bell, and will be related to the generated API docs. I would recommend ignoring it for now and I'll have a closer look once this is ready to go in.

weaverba137 · 2025-12-04T22:14:55Z

@bsipocz, I'm planning to resume work on this PR in the upcoming weeks, in anticipation of a new API version being deployed on the NOIRLab side.

One important question above is about function_name_async functions. I responded to a suggestion with further questions, which I still need answers for.

There is a less urgent question about whether sia_url should be an attribute/property. In NOIRLab's case there are two separate URLs so it kind of has to be a function.

Other open suggestions are things I just need to look up or otherwise implement.

weaverba137 · 2026-01-14T20:37:43Z

@bsipocz, @keflavich, I hope to finish up a few remaining issues with remote tests (which boil down to column ordering in returned data) in the next day or so. There are still two remaining questions from my previous comment. Addressing those should be easy.

weaverba137 · 2026-01-14T20:38:48Z

PS, all of the test failures on this PR currently are due to items outside of my control.

weaverba137 · 2026-01-14T21:59:08Z

I solved a documentation warning issue I mentioned way back in July. All remaining documentation warnings appear to be from other packages.

The ultimate cause of the warning issue was that automodapi creates a certain header hierarchy (i.e. underline style), and the rest of your document has to be consistent with that, otherwise you get a warning about underlining that doesn't even appear in your document. Once you adjust the underlining to match automodapi the warning goes away.

bsipocz · 2026-01-15T03:18:11Z

I haven't tried to run the docs build locally, but these failures look familiar. I would recommend rebasing the whole PR on top of the current main and suspect it may just solve it out of the box.

bsipocz · 2026-01-15T18:37:30Z

RTD is known to fail and will be fixed separately.

weaverba137 · 2026-01-15T18:38:11Z

RTD is known to fail and will be fixed separately.

Thanx. I had already done a recent merge with main.

weaverba137 · 2026-02-03T16:54:49Z

@bsipocz @keflavich , can we move this toward merging? There is one unresolved question about the meaning of "async" above. At this point I am tempted to remove any reference to "async" at all.

bsipocz

This is good to go from my POV, but some minor things need to be fixed before pushing the merge button:

3 remote tests are failing. With a quick look it feels like that it's more about a changed/undetermined column ordering rather than about content differences, maybe it should be enough to compare sets of column names and spot check content of one or two columns rather than full tables.

Other things:

re: async: let's merge this as is, and then return to rename the kwarg with something more descriptive. We would need to do that here, but in other modules as well.
a setuptools incompatibility has bubbled up, so tests likely won't run here at all, but it will be all good once you rebase this.
RTD build should be all OK after merge/rebase

bsipocz · 2026-02-10T23:16:14Z

+        async : :class:`bool`, optional
+            If ``True``, return the raw query response instead of a Table.


I would still think we need to find a better keyword name for this than async. Maybe spell it out as return_raw_query or return_raw_response or similar. But to be constructive and not holding up merging this one I would suggest bumping it for a future PR.

bsipocz · 2026-02-10T23:23:14Z

3 of these tests are failing, would you mind looking into them?

../../.tox/test-online/lib/python3.12/site-packages/astroquery/noirlab/tests/test_noirlab_remote.py::test_query_file_metadata FAILED [ 33%] ../../.tox/test-online/lib/python3.12/site-packages/astroquery/noirlab/tests/test_noirlab_remote.py::test_query_file_metadata_minimal_input FAILED [ 66%] ../../.tox/test-online/lib/python3.12/site-packages/astroquery/noirlab/tests/test_noirlab_remote.py::test_query_hdu_metadata FAILED [100%]

Working on this...

bsipocz · 2026-02-11T00:18:57Z

Thank you!

I'm still going ahead and rebase this to get rid of the merge commits. History is much cleaner that way in case we even need to go back to look at a bisect.

bsipocz · 2026-02-11T00:41:45Z

Huge thanks @weaverba137 for making this happen!

weaverba137 marked this pull request as ready for review July 17, 2025 23:51

bsipocz added New Service noirlab labels Aug 12, 2025

bsipocz added this to the v0.4.11 milestone Aug 12, 2025

bsipocz reviewed Aug 12, 2025

View reviewed changes

bsipocz reviewed Sep 13, 2025

View reviewed changes

bsipocz modified the milestones: v0.4.11, 0.4.12 Sep 18, 2025

bsipocz approved these changes Feb 10, 2026

View reviewed changes

weaverba137 added 25 commits February 10, 2026 16:17

adding more tests

ac0ead5

activate additional tests and add placeholders

5efeb08

fix style issue

991bdb0

working on doc strings

eeb1267

fix pformat issues

e110858

update docs

8dc644f

add test coverage for some corner cases

9705da9

fix style & add placeholders

e1d6387

tweak formatting

24da457

update and sync docs

37cd840

activate hdu metadata tests

b56a52b

all tests now active

66d39d0

fix remote data error

fb6973b

fix doc compilation errors

8df43b7

updates based on initial review

ac78b9e

fix style check

b28e6d6

set up to ignore column prefixes

5bbfd54

update tests with some workarounds

3669068

consolidate some methods

55e43c4

Fixed all remote tests

ec8713c

change header underlining

669d636

clean up some doc warnings

ea0677e

promote sia_url to public method

cbc3a21

Fix references to async_ keyword

ffef537

remove some strict requirements on remote tests

8457192

bsipocz force-pushed the noirlab-archive branch from 155f337 to 8457192 Compare February 11, 2026 00:19

bsipocz merged commit 79e0425 into astropy:main Feb 11, 2026
11 checks passed

weaverba137 deleted the noirlab-archive branch February 11, 2026 17:02

		async : :class:`bool`, optional
		If ``True``, return the raw query response instead of a Table.

Uh oh!

Conversation

weaverba137 commented Jun 26, 2025 • edited by bsipocz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weaverba137 commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weaverba137 commented Jun 26, 2025

Uh oh!

bsipocz commented Jun 27, 2025

Uh oh!

weaverba137 commented Jul 17, 2025

Uh oh!

weaverba137 commented Jul 17, 2025

Uh oh!

bsipocz commented Jul 18, 2025

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

weaverba137 commented Aug 13, 2025

Uh oh!

bsipocz commented Aug 13, 2025

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weaverba137 commented Sep 13, 2025

Uh oh!

weaverba137 commented Sep 17, 2025

Uh oh!

weaverba137 commented Sep 17, 2025

Uh oh!

bsipocz commented Sep 18, 2025

Uh oh!

weaverba137 commented Dec 4, 2025

Uh oh!

weaverba137 commented Jan 14, 2026

Uh oh!

weaverba137 commented Jan 14, 2026

Uh oh!

weaverba137 commented Jan 14, 2026

Uh oh!

bsipocz commented Jan 15, 2026

Uh oh!

bsipocz commented Jan 15, 2026

Uh oh!

weaverba137 commented Jan 15, 2026

Uh oh!

weaverba137 commented Jun 26, 2025 •

edited by bsipocz

Loading

weaverba137 commented Jun 26, 2025 •

edited

Loading

codecov Bot commented Jun 26, 2025 •

edited

Loading

bsipocz Jan 15, 2026 •

edited

Loading