Skip to content

RCA-1039: refactor roman_catalog_process.py to only run informer stage when necessary.#14

Merged
mairanteodoro merged 3 commits intospacetelescope:mainfrom
mairanteodoro:RCAL-1039-informer-changes
Apr 30, 2025
Merged

RCA-1039: refactor roman_catalog_process.py to only run informer stage when necessary.#14
mairanteodoro merged 3 commits intospacetelescope:mainfrom
mairanteodoro:RCAL-1039-informer-changes

Conversation

@mairanteodoro
Copy link
Collaborator

Resolves RCAL-1039

The refactoring of roman_catalog_process.py was intended to prevent running the informer stage unnecessarily. It will check for the existence of a previously created model that contains all the information necessary for the estimation stage (the grid of models), which is where we actually determine the best model. roman_photoz will now check for the presence of a pickle file named "roman_model.pkl" (default) or with the name provided by the user through the --model_filename argument. If the file exists, we'll use it, otherwise, we'll run the informer stage to create one. In either case, the file location is in the $LEPHAREWORK dir.

@mairanteodoro mairanteodoro changed the title Rcal 1039 informer changes RCA-1039: refactor roman_catalog_process.py to only run informer stage when necessary. Apr 18, 2025
@mairanteodoro mairanteodoro marked this pull request as ready for review April 18, 2025 20:06
@mairanteodoro mairanteodoro requested a review from schlafly April 18, 2025 20:06
Copy link
Collaborator

@schlafly schlafly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Can you confirm that if only that pickle file exists, the "estimator" stage would run successfully?

If something like that is true, we should consider putting that pickle file in CRDS and updating it periodically. And temporarily putting that pickle file somewhere like /grp/roman/scsb/roman_photoz_data/roman_model.pkl
where it would get picked up. And then the default mode of operation would be for it to look at a LEPHARE_MODEL environment variable or something to pick up that file; we'd create it with a separate mechanism. Does something like that make sense to you?

@mairanteodoro
Copy link
Collaborator Author

This looks good. Can you confirm that if only that pickle file exists, the "estimator" stage would run successfully?

Yes, there is a unit test for that in this PR (test_process_model_existence()).

@mairanteodoro mairanteodoro merged commit b27e2bc into spacetelescope:main Apr 30, 2025
1 of 2 checks passed
mairanteodoro added a commit to mairanteodoro/roman_photoz that referenced this pull request May 2, 2025
…e when necessary. (spacetelescope#14)

* Implement check for previously created model file.

* Style check fixes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants