Skip to content

Following the Development HEAD

Jump to bottom
Michael McCrackan edited this page May 20, 2022 · 37 revisions

Welcome to the tolteca wiki!

Following the HEAD

TolTEC is in the beta-release and is still in active development.

The tolteca.simu and tolteca.reduce interface is not stable and this page keeps track the latest breaking changes which may require the user to re-clone the repos, re-compile the source code, or re-run the simulation/reduction. Please re-visit this page frequently or if you find any issue.

2022-05-20

A few new features have been added to citlali v1.2.1:

  1. There is a new required config option "runtime, use_subdir." If this is set to true,citlali will act as usual and make the red00, redu01 directories. If false, it will not make a new sub-directory and put the obsnum and coadded directories into the output_dir. Most will likely just want to leave this to true.
  2. The noise map FITS extension names have been fixed as have the PSD and histogram file variables when polarization is enabled.
  3. The units of each map have been added to their respective headers. Conversions between mJy/beam and MJy/Sr are now both in the primary HDU.
  4. There have been some internal updates to how beamsize are handled. This will slightly change the flux conversion (and the resulting fluxes for the signal maps) for the 1.4 mm and 2.0 mm arrays when mJy/beam is selected, but it is now consistent with the input apt table.

2022-05-17

citlali v1.2.1 has been released with the following changes:

  1. The config option "mapmaking, cunit" is now enabled and can accept either "MJy/Sr" or "mJy/beam" as input (these are case-sensitive for now). This will make all maps, psds, histograms, and tables for science and pointing maps in the specified units.
  2. There has been a fix in the conversion between MJy/Sr to mJy/beam, which was a parameter in the primary HDU header.
  3. The config option "timestream, output, chunk_type" is now enabled and accepts either "rtc" or "ptc." This lets you output either a raw timestream (after despike, filter, despike, calibration) or a processed timestream (after PCA cleaning).
  4. We have added beammap specific calibration items for boresight offsets, source photometry, and extinction. These will not affect science/pointing observations and are optional, so if you aren't using beammap mode, you don't need to add them to 70_reduce.yaml file. They are included in the example config.yaml file: https://github.com/toltec-astro/citlali/blob/89f047e50a03322b5eefb94e69740640ec42f8b6/data/config.yaml#L8
  5. Some adjustments to the noise maps have been made, fixing a bug in the map-making stage and in the Wiener filter.
  6. The Wiener filter has been updated. When not using gaussian_template, the output flux now agrees better with expectations (kernel flux is no longer overestimated).
  7. Jinc map filter is now implemented. This can be enabled by changing "mapmaking, method" to jinc. There is still some iteration to be done on the parameters that control the shape of the filter. It is also very slow compared to the naive map-maker and is currently prohibitively slow for large maps. I'm not sure if this can be improved but it is under investigation.
  8. You may run into memory issues on large maps. This does not appear to be a memory leak, but is due to the fact that citlali's average memory usage is higher than anticipated. It isn't clear if this can be alleviated yet, so if you run into memory issues, try increasing the amount of memory you're using.
  9. There is a new map in the output files called coverage_bool_{STOKES_PARAM}. This is a boolean map that is true where weight values are above the coverage_cut and false if they are below. Apply this mask to the signal map to remove outlier values on the edge of the map.
  10. The HDU layer names have been updated. They are now consistent between polarization and non-polarization reductions. This means that for a non-polarized reduction, the signal map layer will now be called "signal_I" instead of "signal."
  11. The average noise PSD and average noise Histogram values have been added to the respective noise PSD and Histogram netcdf files.
  12. The WCS naxis parameters have been updated to have the correct number of axes (4).
  13. The WCS information has been removed from the primary HDU to match with conventions.
  14. The exposure time (t_exptime) has been added to the primary HDU header.

2022-03-28

tolteca v1.2.2 released. In this version a number of issues in the simulator is addressed:

  1. The fluxscale factor. This is now calculated using the detector responsivity directly.
  2. The tune scheme. Previously the tune was done at the center of the observation and each detectors is tuned to the background loading they see. This introduces offsets in power loading zeropoint for the timestreams. In the new version I set this so all the detectors are tuned to the same power loading, which is consistent with the version v0 of the tolteca.simu. I am still learning the implications of this, especially related to the PCA clean.

There is no API change in this version.

2022-02-16

The latest version v1.1.0 is released.

In this version, the API of all RuntimeBase subclasses are cleaned up for better extendability.

Also included in this version is the new tolteca web commandline interface that runs the web-based tools such as the TolTEC Observation Planner on local machine.

Instructions for running the obs planner locally

The obs planner has two versions:

  • A legacy version is (almost) identical to the one runs currently on toltecdr.astro.umass.edu, which is denoted as obs_planner_v0.
  • A WIP version that implements the pre-set templates of mapping patterns, which is denoted as obs_planner.

As of right now, only the legacy version obs_planner_v0 is fully functional.

To run the obs_planner_v0, additional dependency is required:

$ pip install git+https://github.com/toltec-astro/dasha.git@dash2

If any error occurs, please try updating the following packages first:

$ pip install --upgrade git+https://github.com/toltec-astro/tollan.git
$ pip install --upgrade git+https://github.com/toltec-astro/kidsproc.git
$ pip install --upgrade git+https://github.com/toltec-astro/tolteca.git

Also required is the toltec_sensitivity module, which does the sensitivity calculation. This module cannot be installed via pip, but with git only:

$ git clone https://github.com/toltec-astro/toltec_sensitivity.git

Make note of the path of the cloned module repo, as it will be used in the config files we compose next.

The tolteca web commandline interface works similarly to the other commandline interface, where user can supply config YAML files through workdir or use the -c global switch.

Here is an example configuration section that configures the obs_planner_v0 app:

web:
  apps:
  - name: obs_planner_v0
    raster_model_length_max: 1.3 deg
    lissajous_model_length_max: 7 arcmin
    t_exp_max: 1.5 hr
    toltec_sensitivity_module_path: /your/cloned/path/of/toltec_sensitivity
    title_text: "My TolTEC Obs Planner"

Add the above snippet your tolteca workdir as 20_web.yaml, and run tolteca web -a obs_planner_v0 will fire up the web server. Then visit in your browser 127.0.0.1:8050 to access the obs planner web page.

For more general description of the tolteca.web, see the documentation here: https://toltec-astro.github.io/tolteca/tolteca/web.html

2022-02-15

The kernel fits input option to citlali has been updated to allow for the FITS extension for the input kernel image to be selected at runtime. To use this functionality, set kernel, type to image and kernel, filepath to the path and name of the file.

You will need to update your 70_reduce.yaml scripts to add the keyword image_ext_name under the kernel section (see https://github.com/toltec-astro/tolteca/blob/main/tolteca/data/examples/70_redu_simulated_data.yaml for an example). This can be any string that corresponds to an extension name in the input FITS file.

2022-02-02

citlali has been updated so that it will now create sub-directories for each of the data products. Running citlali once will create a sub-directory in the tolteca reduce directory called redu0. Subsequent runs of tolteca reduce will create additional sub-directories called redu1, redu2, etc. The redu sub-directories contain a directory for each obsnum and for the coadded data products if coadding is requested.

These sub-directory names will change in future releases.

2022-01-20

The latest version v1.0.2 has the TOAST atm model re-enabled in the simulator.

2022-01-14

citlali will not compile with the latest version of the Boost library (1.77) when using Clang13 due to c++-20 requirements. To compile, install boost via homebrew and add -DUSE_INSTALLED_BOOST=On when running cmake.

2022-01-12 (NEW VERSION, BREAKING CHANGE)

The tolteca version v1.0.1 is released. The changes includes:

  • The polarimetry simulation
  • Change of simulator config
  • Completed config schema doc

To start using the new simulator, take a look at the example yaml files in https://github.com/toltec-astro/tolteca/tree/main/tolteca/data/examples

The new simulator is invoked with tolteca simu, while the old simulator still works, which is available with tolteca simu0

IMPORTANT: The toast atm is not enabled yet for the new simu. So if you need to use TOAST3 atm fluctuations, run simu0!

To see the full description of all config items, go to the doc folder of the workdir (created by tolteca setup), check 00_config_references.txt.

To migrate from old simulation config, the recommended way is to create new workdir, and copy over the old *_simu.yaml config, and edit it such that it works with the new config schema. Here is a list of items to be edited (not exhaust):

  • simu.instrument

    It now accept polarized: true|false to enable or disable polarimetry simulation

  • simu.mapping

    The type is now one of raster, lissajous, rastajous, double_lissajous, or daisy (daisy is not fully implemented yet).

  • simu.obs_params

    t_exp can no-longer have unit of cy. To use the mapping pattern time as t_exp, set t_exp: null.

  • simu.perf_params

    They are now more relevant than before. Take a look at the 00_config_references.txt for the available items to set.

  • simu.sources

    The toltec_array_loading source is renamed to toltec_power_loading.

    The colname_map and extname_map in image and point_source_catalog sources are removed. To specify the data hdu or data column, use data_cols or data_exts for the catalog and fits image. Example:

    - type: image
  filepath: simu_image.fits
  data_exts:
  - extname: '{{array_name|upper}}_{{stokes_param}}'
    stokes_param: ['I', 'Q']
    array_name: ['a1100', 'a1400', 'a2000']
- type: point_source_catalog
  filepath: simu_catalog.asc
  name_col: 'name'
  pos_cols: ['ra', 'dec']
  data_cols:
  - colname: 'flux_{{array_name}}_{{stokes_param}}'
    stokes_param: ['I', 'Q', 'U']
    array_name: ['a1100', 'a1400', 'a2000']
- type: toltec_power_loading
  atm_model_name: am_q50

2021-11-09 (BREAKING HOTFIX)

The config to enable TOAST atm model is revised. The ??_simu.yaml needs to be updated accordingly for enabling the TOAST atm model.

To enable, edit in the ??_simu.yaml sources list, toltec_array_loading config dict, atm_model_name to toast:

simu:
    <other config entires ...>
    sources:
        - type: toltec_array_loading
          atm_model_name: toast
        - <other sources ...>

2021-11-08 (BREAKING CHANGE)

The tolteca version v0.3.0 is released. The API of the tolteca.reduce is re-worked to allow future extension and custom reduction steps and pipelining at the high level (Python based workflow). This version only support citlali >=1.0.

Also enabled in this version is the TOAST3 (https://github.com/hpc4cmb/toast/tree/toast3) atmosphere fluctuation model.

This changes breaks existing config.yaml files which are detailed below:

tolteca workdir

Old tolteca workdir can still be used. However, newly created workdir from tolteca setup will contain updated config entry schema documentation doc/00_config_dict_references.txt and updated example files doc/*.yaml.

In particular, the example file doc/70_redu_simulated_data.yaml contains extensive inline comments explaining the new reduce config dict.

reduce config dict

The old ??_reduce.yaml file needs to be updated following the schema defined in tolteca.reduce.ReduConfig. For those who reduce the simulated data, the example file doc/70_redu_simulated_data.yaml can be used, after modifying the reduce.inputs[0].path to the simulated data output path (i.e., the simu.jobkey in the ??_simu.yaml file).

TOAST3 ATM fluctuation model

The simulator can produce simulated data with TOAST3 atmosphere fluctuation model. To enable this feature, add in the simu config toast_atm: true under simu key:

simu:
    <other simu config entries>
    toast_atm: true

Note that toltec_array_loading source has to be disabled by commenting out the item in the simu.inputs list.

Python package toast3 has to be installed separately. As of now, we do not have sufficient documentation for end users to follow other than those provided in the its GitHub repo: https://github.com/hpc4cmb/toast/tree/toast3.

The TolTEC unity cluster shared TolTECA software environment is updated and can run the TOAST3 ATM simulation.

2021-10-28

Version v1.x of citlali has been released and is now public. This version is a complete refactor of citlali and integrates the refactored common_utils library now called tula, as well as the CONAN C++ package manager. Please review the README.rst file (https://github.com/toltec-astro/citlali/blob/v1.x/README.rst) in the citlali repository to see the changes to the install instructions and base requirements. Citlali v1.x uses C++-20 and an update to compiler versions (LLVM 13+ or GCC 10+) may be required.

This version also uses a completely new yaml config file, an example of which can be found at https://github.com/toltec-astro/citlali/blob/v1.x/data/config.yaml.

Many features of citlali v1.x are still in the debugging and deployment phase and will continue to be updated and released in the near future as they are completed and issues are fixed.

2021-09-21 (BREAKING CHANGE)

The version v0.2.0 is released with a complete rework of the config loading system and commandline API.

To update, reinstall both tollan and tolteca with --upgrade switch, if installed via pip.

For installation via git clone, do git pull in both tollan repo and tolteca repo.

The tolteca workdir created in previous version will stop work. An error message will be shown with instructions of how to proceed for the migration when you attempt run simu or reduce after updating to v0.2.0

You'll need to recreate the workdir with tolteca setup, and copy over any user config files (60_simu.yaml, 80_reduce.yaml, etc) to the new workdir.

Some of the config keys have been changed:

  • simu.obs_params.f_smp_data is changed to simu.obs_params.f_smp_probing

  • For mapping type of raster, the model parameter t_turnover is renamed to t_turnaround.

2021-08-25

The low level config has been changed in the latest version of citlali for handling multiple calibration items. Existing low level config files generated by tolteca reduce has to be regenerated.

2021-08-11

Compilation of citlali with llvm 12.01 failed due to an invalid option in llvm-ranlib. The option has been removed in the latest version of citlali.

The CCfits.cmake file in citlali has been updated to account for further issues preventing the compiler from finding cfitsio.h. Pull the latest citlali/common_utils/cmake version for this fix.

2021-08-09

The cmake installation step for CCFITS in citlali broke due to an out-of-date version being referenced. Do a git pull in citlali/common_utils/cmake to get the updated version of CCFITS.cmake.

2021-08-03

The tolteca simu command line interface now accepts arguments that overrides the entries set in the YAML config files.

In particular, the new option "exports_only" is added for the simulator to export the mapping pattern as LMT Observation Tool (LMTOT) scripts. Currently, only Lissajous and Raster map are supported. We are working on adding more.

To export, run the follows:

$ tolteca simu --exports_only

This will print the exported LMT OT script on screen, and also write the file named {jobkey}*_exported.lmtot in the output directory.

2021-07-21

Updates and fixes to citlali:

  • Adjusted the order by which variables are read from the apt table within citlali. It will now run correctly with the trimmed apt ecsv table produced by tolteca reduce.

2021-07-20

Updates and fixes to citlali:

  • Fixed pointing errors in citlali that were leading to the final maps being offset from the input WCS pointing.
  • Flipped output maps along the x direction to match input orientation due to CCFITS output formatting.
  • Added in and fixed flux calibration correction to citlali based on values in apt ecsv tables.
  • Added in 'a_fwhm' and 'b_fwhm' to the 'lali' and 'beammap' classes within citlali for kernel maps, which are read from the the apt ecsv tables.
  • A signal-to-noise map has also been added into the output FITS files for science maps only.

2021-07-13

Added a new mapping pattern "Rastajous" to the simulator. The example config can be found here: https://github.com/toltec-astro/tolteca/blob/aeba68d263d25254b65105951c9a0d6e06925759/tolteca/data/examples/60_simu.yaml#L35

Added a_fwhm and b_fwhm to the generated apt from tolteca.simu. Rerunning the simulation is required for citlali to make use of this info to generate more accurate kernel maps.

2021-07-10

Added tolteca command tolteca check that checks the installation and configuration and outputs a summary for potential issues and the fix.

Added a new mapping pattern "DoubleLissajous" to the simulator. The example config can be found here: https://github.com/toltec-astro/tolteca/blob/aeba68d263d25254b65105951c9a0d6e06925759/tolteca/data/examples/60_simu.yaml#L35

2021-06-29

The citlali has been updated in various places, including enable handling the proper conversion of timestream to physics unit internally. The apt_*_trimmed.ecsv files generated by the tolteca.reduce module need to be re-generated. This will happen automatically if you run citlali with tolteca reduce, but will not be done if run directly the citlali executable with the generated low-level config.

Some of the low level config has been updated, you'll need to revise the reduce YAML config (check the updated 80_reduce.yaml here: https://github.com/toltec-astro/tolteca/blob/main/tolteca/data/examples/80_reduce.yaml):

  • removed map.pattern
  • added map.timechunk
  • update pcaclean.approximateWeights type to bool.

2021-06-28

The SPDLOG upstream has updated and breaks the citlali. If you found any compilation error, please run git pull && git submodule update --recursive to update the repo, delete the relevant build dependencies rm -r build/_deps/spdlog-* and re-run cmake.

Clone this wiki locally