RCA-1039: refactor roman_catalog_process.py to only run informer stage when necessary.#14
Conversation
schlafly
left a comment
There was a problem hiding this comment.
This looks good. Can you confirm that if only that pickle file exists, the "estimator" stage would run successfully?
If something like that is true, we should consider putting that pickle file in CRDS and updating it periodically. And temporarily putting that pickle file somewhere like /grp/roman/scsb/roman_photoz_data/roman_model.pkl
where it would get picked up. And then the default mode of operation would be for it to look at a LEPHARE_MODEL environment variable or something to pick up that file; we'd create it with a separate mechanism. Does something like that make sense to you?
Yes, there is a unit test for that in this PR ( |
…e when necessary. (spacetelescope#14) * Implement check for previously created model file. * Style check fixes.
Resolves RCAL-1039
The refactoring of
roman_catalog_process.pywas intended to prevent running the informer stage unnecessarily. It will check for the existence of a previously created model that contains all the information necessary for the estimation stage (the grid of models), which is where we actually determine the best model.roman_photozwill now check for the presence of a pickle file named"roman_model.pkl"(default) or with the name provided by the user through the--model_filenameargument. If the file exists, we'll use it, otherwise, we'll run the informer stage to create one. In either case, the file location is in the$LEPHAREWORKdir.