Final Project: Estimating Astronomical Distances via Photos

Photometric redshift pipeline with, primarily, luigi and tensorflow. See docs link on the right for a more descriptive overview. Currently refactoring with poetry and some different design choices.

Note: For downloading the tabular data, I recommend adding a .env file with your AWS secrets like so:
AWS_ACCESS_KEY_ID=something
AWS_SECRET_ACCESS_KEY=something

The downloaded tabular data is 2.9MB.

However, you can also download the file here (preserve the file name or make sure that final_project/conf/aws_paths.yaml tabular_data entry matches the file name you chose).
If you place the file into a folder called 'data' off the main directory of the repo, it will run without trying to download the file from AWS.

The SQL query used to grab the data via CasJobs:

SELECT TOP YOUR_LIMIT_HERE #(10000 for the google drive copy)
  za.specObjID, za.bestObjID, za.class, za.subClass, za.z, za.zErr,
  po.objID, po.type, po.flags, po.ra, po.dec,
  po.run, po.rerun, po.camcol, po.field,
  po.cModelFlux_u, po.cModelFlux_g, po.cModelFlux_r, po.cModelFlux_i, po.cModelFlux_z,
  (po.petroMag_r-po.extinction_r) as dered_petro_r,
  zp.z as zphot, zp.zErr as dzphot, zi.subclass as zsubclass,
  zi.e_bv_sfd,zi.primtarget, zi.sectarget,zi.targettype,zi.spectrotype
INTO MyDB.pasquet_2018_with_mag
FROM SpecObjAll as za
    JOIN PhotoObjAll as po ON (po.objID = za.bestObjID)
    JOIN Photoz as zp ON (zp.objID = za.bestObjID)
    JOIN galSpecInfo as zi ON (zi.SpecObjID = za.specObjID)
WHERE
    (za.z < 1 and za.z > 0 and za.zWarning=0)
    and (za.targetType ='SCIENCE' and za.survey='sdss')
    and (za.class='GALAXY' and zi.primtarget>=64)
    and (po.clean=1 and po.insideMask=0)
    and ((po.petroMag_r - po.extinction_r) <= 17.8)
ORDER BY RAND(10);

To run, run pipenv install then pipenv run python -m final_project to kick off the process! Keep note that this will download ~100GB of data (can always interrupt the process), so be wary of storage constraints. For the refactor that's in progress, the number of final galaxy images desired will be configurable as the tabular data will be generated by running the sql query against the SDSS API as part of the pipeline. If you're re-running the same sql query to get the tabular data, it will instead download a cached copy from an S3 bucket.