V0.0.6 dev (#132)

* update stimela in pyproject.toml

* add 3.12 to classifiers in pyproject.toml

* add jax version of psf convolve

* modify kclean to work with undersized psf

* fix wsum normalisation in subminor. avoid circular import. lower kclean test threshold

* clean up external masking logic in kclean

* report which threading layer was in use for numba after sara

* add multi-stokes predict

* don't double normalise wgt per band in L2-reweight steps

* Add joint correlation gridding (#131)

* start adding joint polarisation products

* compute natural grad with jax in hci

* attempt to limit number of jax cpu threads per process

* don't limit XLA cpu device count

* use first element of tuple produced by cg in hci

* tweaks

* get grid app working with joint pol products. Add beams table to fits hdu

* try IFT pol model with approx likelihood

* convolving correlations is hard

* polfusion

* fiddle with BEAMS subtable

* add some checks

* Fix BEAMS subtable so that beam displays correctly in CARTA

* small tweaks to parser

* fix failing tests

* use 90-PA in add_beampars

* reinstate nearsest neighbour gridding for weighting tests

* Also test FS in test_polproducts

* fix flag ncorr for averaging routines, update Dockerfile, remove spotless cab

* cache MFS PSFPARS in dds, make psfpars_mfs optional in dds2fits

* simple transient injection in hci worker (no pol parameters)

* update restore worker to work with multiple times and correlations

* fix typos in restore worker, remove some unused imports

* add complex convolve test when gridding correlations directly

* render wsclean models to mds

* remove product from fits name in model2comps

* typo

* fix axis order in model2comps, write out cube as sanity check

* fix field delection in degrid worker

* do not drop ncorr on flag before weight_data in hci

* add inject-transients to hci config

* pass mask into counts to weigts in hci

* start adding polclean functionality. Don't double normalise restored image by wsum

* add fskclean test

---------

Co-authored-by: Landman Bester <lbester@com08.science.kat.ac.za>

* transpose stokes axis in hci worker

* convert beam params to degrees and add during header construction

* add psf-relative-size to hci. only make it big enough to fit the main lobe by default

* cast PA to degrees in fits header

* also for MFS beam pars

* redefine pa as 90-pa for consistency with fits conventions

* filter extreme counts in hci worker

* mimic no-use-weights-as-taper

* allow single precision weighting

* noop when filter-counts-level is None

* only grid half the uv-plane inside weighting functions

* change uv bins in weighting code, use floor instead of round

* check bounds in both compute_counts and counts_to_weights

* keep uvw and freq in double precision

* add better tests for weighting code. don't use fatmath when computing weights. tidy up and add min_padding parameter to hci worker

* add option to write out weight grid in hci worker

* add arbitrary stokes outputs

* make zarr output of hci worker compatible with what is expected in breifast

* add NAXIS keywords to fits headers. update product checks in init

* fix indentation in hci

* use log.info instead of file=log

* fix failing tests

* reverse sign of X in hci output

* do not use fastmath in wavelets

* init_factor -> init-factor

* log.info -> print in utils

* normalise by wsum when writing zarr in hci. remove manual encoding

* base divergence check on whether both rmax and rms increase, not just rms

* add header ordering in hci comment. fix weighting when k>0

* enforce proper ordering of header parameters when converting WCS to a header

* convert pa to degrees in stokes2im

* update pyscilog

* pyscilog master -> main

* resort to nearest neighbour gridding to compute the weights if all else fails

* divorce pyscilog. start adding rephasing to hci worker

* add logging.py

* update actions and set up dependabot

---------

Co-authored-by: Landman Bester <lbester@com08.science.kat.ac.za>

Author: landmanbester

.github/dependabot.yml

@@ -0,0 +1,33 @@
+version: 2
+updates:
+  # Monitor GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "09:00"
+    open-pull-requests-limit: 10
+    reviewers:
+      - "@bester"
+    assignees:
+      - "@bester"
+    commit-message:
+      prefix: "ci"
+      include: "scope"
+
+  # Monitor Python dependencies
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "09:00"
+    open-pull-requests-limit: 10
+    reviewers:
+      - "@bester"
+    assignees:
+      - "@bester"
+    commit-message:
+      prefix: "deps"
+      include: "scope"

.github/workflows/ci.yml

@@ -11,7 +11,7 @@ on:
   pull_request:
 
 env:
-  POETRY_VERSION: 1.5
+  POETRY_VERSION: 1.8.4
 
 jobs:
   should-run:
@@ -82,24 +82,24 @@ jobs:
         python-version: ["3.10", "3.11", "3.12"]
 
     steps:
+      - name: Checkout source
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version}}
 
       - name: Install poetry
-        uses: abatilo/actions-poetry@v2
+        uses: abatilo/actions-poetry@v3
         with:
           poetry-version: ${{ env.POETRY_VERSION }}
 
       - name: Check poetry install
         run: poetry --version
 
-      - name: Checkout source
-        uses: actions/checkout@v3
-        with:
-          fetch-depth: 0
-
       - name: Restore repo times
         uses: chetan/git-restore-mtime-action@v2
 
@@ -113,7 +113,7 @@ jobs:
           echo "timestamp=$(/bin/date -u '+%Y%m%d%H%M%S')" >> $GITHUB_OUTPUT
 
       - name: Cache Numba Kernels
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           key: numba-cache-${{ matrix.python-version }}-${{ steps.numba-key.outputs.timestamp }}
           restore-keys: numba-cache-${{ matrix.python-version }}-
@@ -124,7 +124,7 @@ jobs:
 
       - name: Load cached CASA Measures Data
         id: load-cached-casa-measures
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           key: casa-measures-${{ hashFiles('measures_dir.txt')}}
           path: |
@@ -163,7 +163,7 @@ jobs:
           python-version: "3.10"
 
       - name: Install poetry
-        uses: abatilo/actions-poetry@v2
+        uses: abatilo/actions-poetry@v3
         with:
           poetry-version: ${{ env.POETRY_VERSION }}
 
@@ -177,7 +177,7 @@ jobs:
         run: poetry build
 
       - name: Publish distribution to PyPI
-        uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+        uses: pypa/gh-action-pypi-publish@v1.12.2
         with:
           user: __token__
           password: ${{ secrets.PYPI_API_TOKEN }}

Dockerfile

@@ -13,5 +13,4 @@ RUN apt -y update && \
 
 RUN python -m pip install -U pip setuptools wheel && \
     python -m pip install -U pfb-imaging@git+https://github.com/ratt-ru/pfb-imaging@main && \
-    python -m pip install numpy==1.22 && \
     python -m pip cache purge

README.rst

@@ -11,8 +11,6 @@ Install the package by cloning and running
 
 :code:`$ pip install -e pfb-imaging/`
 
-Note casacore needs to be installed on the system for this to work.
-
 You will probably may also need to update pip eg.
 
 :code:`$ pip install -U pip setuptools wheel`
@@ -94,6 +92,6 @@ Documentation for each worker is listed under
 Acknowledgement
 ~~~~~~~~~~~~~~~~~
 
-If you find any of this useful please cite (for now)
+If you find any of this useful please cite
 
-https://arxiv.org/abs/2101.08072
+https://arxiv.org/abs/2412.10073

pfb/__init__.py

@@ -9,7 +9,10 @@ def set_envs(nthreads, ncpu):
     os.environ["VECLIB_MAXIMUM_THREADS"] = str(nthreads)
     os.environ["NPY_NUM_THREADS"] = str(nthreads)
     os.environ["NUMBA_NUM_THREADS"] = str(nthreads)
-    os.environ["JAX_PLATFORMS"] = 'cpu'
+    # os.environ["JAX_PLATFORMS"] = 'cpu'
+    # os.environ["XLA_FLAGS"] = ("--xla_cpu_multi_thread_eigen=true "
+    #                            "--xla_force_host_platform_device_count=4 "
+    #                            f"intra_op_parallelism_threads={str(nthreads)}")
     os.environ["JAX_ENABLE_X64"] = 'True'
     # this may be required for numba parallelism
     # find python and set LD_LIBRARY_PATH
@@ -52,14 +55,14 @@ def set_client(nworkers, log, stack=None, host_address=None,
     host_address = host_address or os.environ.get("DASK_SCHEDULER_ADDRESS")
     if host_address is not None:
         from distributed import Client
-        print("Initialising distributed client.", file=log)
+        log.info("Initialising distributed client.")
         if stack is not None:
             client = stack.enter_context(Client(host_address))
         else:
             client = Client(host_address)
     else:
         from dask.distributed import Client, LocalCluster
-        print("Initialising client with LocalCluster.", file=log)
+        log.info("Initialising client with LocalCluster.")
         dask.config.set({
                 'distributed.comm.compression': {
                     'on': True,
@@ -80,7 +83,7 @@ def set_client(nworkers, log, stack=None, host_address=None,
 
     client.wait_for_workers(nworkers)
     dashboard_url = client.dashboard_link
-    print(f"Dask Dashboard URL at {dashboard_url}", file=log)
+    log.info(f"Dask Dashboard URL at {dashboard_url}")
 
     return client