Tests of Gen3 at NERSC

This page describes the set of tests that are being run on DC2 data at NERSC to understand how to work with the gen3 butler.

Prerequisites

DMstack

Currently using DMstack w_2020_27, with plans to move to the next DM weekly where the issues with adding packages to the conda environment have been fixed. This version of DMstack is available via CVMFS which can be set up via source /global/common/software/lsst/cori-haswell-gcc/stack/setup_cvmfs_w_2020_27.sh

LSSTDESC/gen3_workflow

Clone a copy of this repo and use Dominique's u/boutigny/gen3_tests branch

PostgreSQL

Please contact Joanne or Heather to obtain write access to the trial PostgreSQL instance at NERSC. Read-only access is able available and can be set up on your own by running the following in a Cori shell window: /global/common/software/lsst/dbaccess/gen3_reader.sh

Test Setup

Following on from the work at CC-IN2P3, concentrate on a subset of Run2.2i containing 10 visits per filter, all corresponding to tract 3078, patch (4,5). https://github.com/LSSTDESC/gen3_workflow/wiki/Tests-of-gen3-on-DC2#test-setup

A gen2 repository has been created with links to the original Run2.2i production repository: /global/cscratch1/sd/desc/DC2/gen3/Run2.2i-gen2

Creation of gen3 Butler

To avoid the use of the public schema in PostgreSQL we will create our own schema first.

psql -h nerscdb03.nersc.gov -U desc_dm_gen3_admin desc_dm_gen3
CREATE SCHEMA lsst_27;
SELECT * from pg_catalog.pg_namespace;

Create a new file named butler.yaml which includes the name of the new schema just created:

datastore:
  checksum: false
  cls: lsst.daf.butler.datastores.posixDatastore.PosixDatastore
  records:
    table: posix_datastore_records
  root: <butlerRoot>
registry:
  db: postgresql://nerscdb03.nersc.gov:5432/desc_dm_gen3
  namespace: lsst_27

Note we are accelerating the ingest by setting checksum==false.

Now we can create the gen3 butler by first setting up DMstack in our environment and running at the command line:

butler create --seed-config butler.yaml Run2.2i-gen3 where Run2.2i-gen3 is the root location of this gen3 butler within the current directory.

Next register the name of the obs_lsst imsim instrument

butler register-instrument Run2.2i-gen3 lsst.obs.lsst.LsstImSim

Note that the name lsst.obs.lsst.LsstImSim will change, likely being set to LSST-ImSim in upcoming DMstack weeklies.

Temporary bootstrap step which is necesssary to ingest calibration data

Dominique has written an updated version of Jim Bosch's bootstrap script which works for Run2.2i data. We will be using this until calibration data can be ingested in gen3. Obtain a copy of the current bootstrapy.py from Dominique's branch on this repo. Edit your copy of bootstrap.py to update the location of your Run2.2i gen2 repo stored in GEN2_RAW_ROOT. Then run:

./bootstrap.py $PWD/Run2.2i-gen3

It is possible to ingest raws directly via gen3 and this still needs to be tested. Will be done with the next stable DMstack weely.

Ingesting skymaps

Now we can use DM's gen3 tools to ingest a skymap, using the configuration Dominique set up in the gen3_workflow package:

makeGen3Skymap.py Run2.2i-gen3 skymaps/imsim -C $PWD/gen3_workflow/python/desc/gen3_tests/makeSkyMap.py

Ingest reference catalogs

Currently the reference catalog ingest is handled by another script that Dominique wrote in gen3_workflow named ingestRef.py. This script must be edited to point to your local gen2 and gen3 repositories.

python ingestRef.py

processCcd

See Dominique's notes for details, where we set up a yaml to run the appropriate steps to complete processCcd.

Create a new yaml file, named pipelines/ProcessCcd.yaml, for the configuration:

description: ProcessCcd
instrument: lsst.obs.lsst.LsstImSim
tasks:
  isr: lsst.ip.isr.IsrTask
  charImage: lsst.pipe.tasks.characterizeImage.CharacterizeImageTask
  calibrate: lsst.pipe.tasks.calibrate.CalibrateTask

First we initialize the output area and then proceed to run visit by visit:

pipetask run -d "visit = 13289"  -b Run2.2i-gen3/butler.yaml -i "LSST-ImSim/calib,LSST-ImSim/raw/all,refcats,skymaps/imsim" -o desc/demo --register-dataset-types -p pipelines/ProcessCcd.yaml --instrument lsst.obs.lsst.LsstImSim --init-only 

now that the initialization is complete, we can process visit by visit:

pipetask run -d "visit = 13289" -j 25 -b Run2.2i-gen3/butler.yaml --output desc/demo -p pipelines/ProcessCcd.yaml --instrument lsst.obs.lsst.LsstImSim

coaddition

Referring to Dominique's notes concerning coaddition, we set up another configuration yaml:

description: Coaddition
instrument: lsst.obs.lsst.LsstImSim
tasks:
  makeWarpTask:
    class: lsst.pipe.tasks.makeCoaddTempExp.MakeWarpTask
    config:
      matchingKernelSize: 29
      makePsfMatched: True
      modelPsf.defaultFwhm: 7.7
      doApplyExternalPhotoCalib: False
      doApplyExternalSkyWcs: False
      doApplySkyCorr: False
      doWriteEmptyWarps: True
      python: config.warpAndPsfMatch.psfMatch.kernel['AL'].alardSigGauss = [1.0, 2.0, 4.5]
  assembleCoadd: lsst.pipe.tasks.assembleCoadd.CompareWarpAssembleCoaddTask

Then running coaddtion:

pipetask run -d "tract = 3078" -j 25 -b Run2.2i-gen3/butler.yaml  -o desc/demo --register-dataset-types -p pipelines/Coaddition.yaml --instrument lsst.obs.lsst.LsstImSim