Flux Suppression Simulations

[kwazzi-jack]

Hi all, so I set up a script to convert model arrays to fits images. I had not even gotten to recovered flux arrays when I spotted the issue.

Kat7 "True" model data - 2 sources
Looks fine and probably why it did not look so bad in our results.

Meerkat "True" model data - 2 sources
One of the sources isn't even in the image, let alone the fact that one is not at the centre.

VLA-B "True" model data - 2 sources
Both sources are definitely not in the correct place.

This is only the true model image made from the skymodels. Obviously the gridding from the skymodels is not working, so instead of showing the recovered flux fits images and trying to fix this conversion problem, I think I will just follow what you said @landmanbester and create the array myself with no use of skymodels. That way I can ensure all sources are accounted for and I can fix the image size, plus ensure no overlapping sources. Am I mistaken with any of this?

**Edit: I've yet to put the skymodels over the image, but I don't know if that is possible with radio-padre and still trying to get X11 to port to my desktop for tigger.

[landmanbester]

Clearly something is going very wrong. Try divorcing from sky models and just placing pixels on a grid as suggested. Then make the dirty and model images and check that they look reasonable. How are you selecting your cell sizes? You need to make sure that your pixels are small enough to satisfy Nyquist

[o-smirnov]

Yeah something is very wrong with the simulation then. Also, I suggest making images oversampled in the usual way (~5 pixels across the PSF) so that the sources are more obvious.

[kwazzi-jack]

So I've been somewhat successful at switching to generating a 2D model array (being careful not to put sources near the edge) and then using the same cell-size and pixel-size selection process we were using for the fluxtractor script (I.e. satisfies nyquist and used good_size function). Everything seems to be working well, using dirty2ms instead. The new data columns themselves seem great as I can image with wsclean to get a decent image returned, but I seem to be having an issue with ms2dirty. It takes the same parameters as I would when I used dirty2ms. Here are the resulting image plots and flux per source after using ms2dirty.

The "Before" model is the one I construct before feeding into dirty2ms and the rest are the corresponding data columns in my ms fed into ms2dirty. Also, this is all where 100% of the sources is modelled. For 1 and 2 sources case, the auto selection of cell- and pixel-size seems to be fine for all arrays. I see for some reason that it does not work for the 100 source case. The model image before fits all sources but the images after do not fit all of them. They use the same cell-size and pixel-size so I'm not sure what could be wrong here.

KAT7 - 1 source

KAT7 - 2 sources

VLA-B - 1 source

VLA-B - 2 sources

VLA-B - 100 sources

MeerKAT - 1 source

MeerKAT - 2 sources

MeerKAT - 100 sources

[landmanbester]

Just be careful with your terminology. Model images are in units of Jy/pixel, dirty images are in units of Jy/beam only after you divide them by the sum of the weights (usually called wsum). So none of the images you show are in the correct units.

As for the 100 source dirty images not making sense, are you just looking up pixel values at the corresponding sources or are you actually running the flux extraction script on them?

[kwazzi-jack]

I think that might be the issue here then. I have not run it through the fluxtractor script yet. But I think I will do so and see what happens. Also do I need to pass a mask when I create my data columns from ms2dirty?

[landmanbester]

Yeah that will make a difference but I am not sure if that is all that is wrong because you still expect to get roughly the correct flux at the locations of the brightest sources (once you divide by wsum). The mask in ms2dirty is actually the negation of the flags not the image space mask that you use for source extraction. You don't have to worry about it unless you have flagged data

[kwazzi-jack]

So I ran the pipeline on the new data simulation. I decided to write a rough outline of my pipeline so you can see if there are any issues and to also help fill in what I've been doing.

Parameters

Here is the params governing the pipeline:

• Arrays: KAT7, VLA-B and MeerKAT
• Sources: 1, 2, and 100 (KAT7 only run on 1 and 2)
• % modelled sources:
  • Source = 1: [100%]
  • Source = 2: [100%, 70%]
  • Source = 100: [100%, 90%, 80%, 70%]
• Noise ~ CN(0, 4)
• Dimensions:
  • n_time = 720
  • n_ant = 7, 27, 64 (respectively)
  • n_chan = n_dir = n_corr = 1
• Scan parameters:
  • length = 2 hr
  • dt = 10 s
  • freq_0 = 1 GHz
• Gains:
  • Amplitude-only
  • Variability is moderate-to-high (not sure how to phrase this)
• Sky-model:
  • Generated from 2D image
  • FOV:
    • Source = 1 and 2: FOV = 0.2'
    • Source = 100: FOV = 0.5'
  • Pareto alpha = 2
  • Peak flux (brightest source) = 1.0 Jy
  • Buffer zone around edge:
    • Source = 1 and 2: 1 pixel
    • Source = 100: 5 pixels
• kalcal:
  • Process Noise (log-scale) = 1e-6 to 1e-1
  • Measurement Noise = Same as noise level
• quartical
  • Time intervals (linear-scale) = 1 to 50
  • Weights (sigma column) = Same as noise level

Simulation

Simulation is done as followed:

• Generate ms for kat7, vlab, and meerkat using simms
• Generate true-gains using above ms to be used for corruption
• Create model images based on auto-selected FOV, cell- and pixel-size with given number of sources, where dynamic range of the flux is determined using a pareto distribution, where we reduce % modelled source flux by removing the faint sources till we reach a total flux equal to the required % modelled source flux. Sources are single, unique pixels in the image and positions are also saved. Buffer zone inserted to ensure no sources are placed near the edge.
• Generate model-vis using the model images above via dirty2ms
• Corrupt model-vis with true-gains from above with africanus corrupt-vis to get clean-vis
• Add noise to clean-vis to get our vis
• Save model-vis, clean-vis, and vis to our ms

Calibration

Calibration algorithms used are:

• kalcal:
  • ekf run once over data, then eks over ekf results
  • fixed noise parameter (sigma_n) to that of simulated data
  • calibrated over process noise using log-space
  • gains saved in codex-format (with extra dimension for augmented complex part) = (n_time, n_ant, n_chan, n_dir, n_corr, n_aug)
• quartical:
  • Solve for G term over 25 iterations
  • Diag-complex mode
  • No model-vis recipe, use simulated model-vis from before
  • Set weights according to sigma-column
  • Gains saved and converted to codex-format = (n_time, n_ant, n_chan, n_dir, n_corr)

Imaging

Perform single pixel per source imaging using:

• Reduced, direct version of pcg algorithm used in pfb-clean
• Hessian comprising of going from image to vis to image while applying a mask created using our simulated model source positions, and weighting associated with calibrated gains used above
• Feed into pcg to get recovered image
• Using model positions, extract flux at given source pixel positions
• Save fluxes in original order of brightest source to faintest
• Same grid for kalcal and quartical is used when imaging the calibrated gains from calibration

Plotting results

We plot the following results:

• Source = 1 and 2:
  • Gains mse for kalcal over process noise (against optimal quartical result)
  • Gains mse for quartical over time interval (against optimal kalcal result)
  • Flux ratio per source (only implemented to show brightest, modelled source for now)
• Source = 100
  • Gains mse for kalcal over process noise (against optimal quartical result)
  • Gains mse for quartical over time interval (against optimal kalcal result)
  • Average % error over both kalcal and quartical
  • Weighted average over both kalcal and quartical (as a check)

Results

The MSE on the gains seems appropriate as before:

1.1 Gains MSE: kalcal over process noise (against quartical), single source

1.2 Gains MSE: quartical over time intervals (against kalcal), single source

2.1 Gains MSE: kalcal over process noise (against quartical), two sources

2.2 Gains MSE: quartical over time intervals (against kalcal), two sources

3.1 Gains MSE: kalcal over process noise (against quartical), 100 sources with MeerKAT

3.2 Gains MSE: quartical over time intervals (against kalcal), 100 sources with MeerKAT

4.1 Gains MSE: kalcal over process noise (against quartical), 100 sources with VLA-B

4.2 Gains MSE: quartical over time intervals (against kalcal), 100 sources with VLA-B

5.1 Flux Ratio: kalcal vs quartical, one source

5.2 Flux Ratio: kalcal vs quartical, two sources

6.1 Average % Error: kalcal vs quartical, 100 sources with MeerKAT

6.2 Average % Error: kalcal vs quartical, 100 sources with VLA-B

7.1 Weighted average: kalcal vs quartical, 100 sources with MeerKAT

7.2 Weighted average: kalcal vs quartical, 100 sources with VLA-B

Notes

• kalcal:
  • Has to use a slightly smaller sigma_f to achieve optimal results (which I thought would be much larger? Like a magnitude less at least).
  • Avg. % error makes sense as well, but I do not understand the slight dip on the left, but it could be the grid I am using possibly?
  • MSE makes sense, could use a smaller grid size for sigma_f
  • Across all plots, the values of the metrics only change slightly as % modelled sources decreases
• quartical:
  • Gains MSE makes more sense now and behaves like it should.
  • The rest of the plots do not make sense to me, again, performing too well on any time interval, even when it is really small.

[kwazzi-jack]

This is the same issue as before, sadly and I am not quite sure what the to do. I decided to check out the result after switching quartical for cubical. It would be known to be worse than quartical, and it was set to mimic quartical as close as possible. Only thing I did forgot about was setting weights for cubical to the noise value of the data, which I'm not sure how much it would impact the results? Anyway, here is the resulting plots, similar to those above:

1.1 Gains MSE: kalcal over process noise (against cubical), single source

1.2 Gains MSE: cubical over time intervals (against kalcal), single source

2.1 Gains MSE: kalcal over process noise (against cubical), two sources

2.2 Gains MSE: cubical over time intervals (against kalcal), two sources

3.1 Gains MSE: kalcal over process noise (against cubical), 100 sources with MeerKAT

3.2 Gains MSE: cubical over time intervals (against kalcal), 100 sources with MeerKAT

4.1 Gains MSE: kalcal over process noise (against cubical), 100 sources with VLA-B

4.2 Gains MSE: cubical over time intervals (against kalcal), 100 sources with VLA-B

5.1 Flux Ratio: kalcal vs cubical, one source

5.2 Flux Ratio: kalcal vs cubical, two sources

6.1 Average % Error: kalcal vs cubical, 100 sources with MeerKAT

6.2 Average % Error: kalcal vs cubical, 100 sources with VLA-B

7.1 Weighted average: kalcal vs cubical, 100 sources with MeerKAT

7.2 Weighted average: kalcal vs cubical, 100 sources with VLA-B

Not too sure what to make of this either, but it does seem to show the avg. % error to worsening as time interval decreases, unlike with quartical. On a side note, is using a single correlation not made the problem less complex, resulting in solutions that are far easier to reach and much closer to the true ones?

Let me know if you want to see the fits images as well of the results.

[kwazzi-jack]

Found a single bug through imaging the data columns with the 2 source case where I had overwritten some visibility values reducing it something similar to the single source case. The 1 source and 100 source case should be fine.

[landmanbester]

Wow that's a lot of information to take in. At least things are making more sense now. The simulation you describe seems sensible. I am surprised by how big the difference between CubiCal and QuartiCal is. They should be pretty much equivalent. I guess Jon but some magic sauce in the QuartiCal recipe. Some things to check:
i) Are you interpolating the qcal gain solutions to full resolution before doing the flux extraction or are you applying a sum of boxcars? If the former it may explain why qcal is doing so well on the flux ratio plots (5.1 and 5.2). The CubiCal plots look more like I expect.
ii) Why are the kalcal results so different for the average % error between the two runs (6.1. and 6.2)? Are you not using the same parameters for kalcal between the two runs? The results for the second run look more like I expect.
iii) I don't think the weighted sum is a good metric to show in your thesis because it does not provide information on individual sources but rather some kind of global average. I think this can be misleading. You can put this in your thesis as a motivation for looking at the average % error instead

[kwazzi-jack]

Well with regards to the differences between quartical and cubical, that is where I am not sure if they are "exactly" the same based on the configuration files.

i) Are you interpolating the qcal gain solutions to full resolution before doing the flux extraction or are you applying a sum of boxcars? If the former it may explain why qcal is doing so well on the flux ratio plots (5.1 and 5.2). The CubiCal plots look more like I expect.

I am applying as a sum of boxcars, by extending the solutions as constant over the specified time-interval index in time. I do this for both quartical and cubical.

ii) Why are the kalcal results so different for the average % error between the two runs (6.1. and 6.2)? Are you not using the same parameters for kalcal between the two runs? The results for the second run look more like I expect.

They use the same parameters. The only difference would be the arrays used (so VLA-B or MeerKAT). I'm not sure what is happening for sigma_f < 1e-5 in 6.1 with MeerKAT. The only other notable differences would be cell-size, image-size and simulated data.

iii) I don't think the weighted sum is a good metric to show in your thesis because it does not provide information on individual sources but rather some kind of global average. I think this can be misleading. You can put this in your thesis as a motivation for looking at the average % error instead

I am aware of this as we discussed before. I'm using it as a test metric only. I understand how the weighted average should look and so if it deviates from this then I know there is an issue and most probably it is affecting the other results as well.