Using casacore measures for computing parallactic angles acquires/drops the GIL excessively

[JSKenyon]

• Codex Africanus version: 0.3.2
• Python version: 3.8.12
• Operating System: Pop!_OS 18.04

Description

When utilising the beam inside the fused RIME (in QuartiCal), performance takes a massive hit. @ulricharmel is seeing 60+ hour runtimes with the beam compared to around 8 without. Is this the same issue as the parallactic angle computation i.e. is the GIL to blame?

[sjperkins]

This is not surprising, the beam is sampled twice per row, and there is a significant amount of computation involved: https://github.com/ska-sa/codex-africanus/blob/174645637b39ea037a7b557e29cdebd55ba0146c/africanus/experimental/rime/fused/terms/cube_dde.py#L149-L295

The performant solution would to evaluate antenna terms per unique, time, feed and antenna, multiply them together and cache the results for use within the RIME onion.

[sjperkins]

Is this the same issue as the parallactic angle computation i.e. is the GIL to blame?

Also, I still believe from our discussions at the workshop that the use of the garbage collector in Quartical leads to poor performance. Has this changed?

[JSKenyon]

Also, I still believe from our discussions at the workshop that the use of the garbage collector in Quartical leads to poor performance. Has this changed?

I think that I had come to an inaccurate conclusion due to the dataset in question containing several fields and correspondingly varied xds sizes. While I will make an effort to confirm that the garbage collector isn't to blame, I am pretty sure something else is afoot. Does the beam automatically compute the parallactic angle or does it rely on the user enabling it?

[sjperkins]

Also, I still believe from our discussions at the workshop that the use of the garbage collector in Quartical leads to poor performance. Has this changed?

I think that I had come to an inaccurate conclusion due to the dataset in question containing several fields and correspondingly varied xds sizes. While I will make an effort to confirm that the garbage collector isn't to blame, I am pretty sure something else is afoot. Does the beam automatically compute the parallactic angle or does it rely on the user enabling it?

The above code does not compute parallactic angles, but it does consume parallactice angles computed by a transformer which computes them per unique time, feed and antenna.

[sjperkins]

Both the Beam and Feed Rotation get their parallactic angles from a common source:

https://github.com/ska-sa/codex-africanus/blob/174645637b39ea037a7b557e29cdebd55ba0146c/africanus/experimental/rime/fused/transformers/parangle.py#L36-L84

[JSKenyon]

This line:
https://github.com/ska-sa/codex-africanus/blob/174645637b39ea037a7b557e29cdebd55ba0146c/africanus/experimental/rime/fused/transformers/parangle.py#L47
calls
https://github.com/ska-sa/codex-africanus/blob/174645637b39ea037a7b557e29cdebd55ba0146c/africanus/experimental/rime/fused/transformers/parangle.py#L29-L34
which I believe means that the beam implementation will also exhibit the same problems as enabling the parallactic angle rotation. I may, however, be completely mistaken.

[JSKenyon]

I do have a decent test now where I have all the necessary beams etc, so I will make sure that the GC isn't involved.

[sjperkins]

This line:

https://github.com/ska-sa/codex-africanus/blob/174645637b39ea037a7b557e29cdebd55ba0146c/africanus/experimental/rime/fused/transformers/parangle.py#L47

calls

https://github.com/ska-sa/codex-africanus/blob/174645637b39ea037a7b557e29cdebd55ba0146c/africanus/experimental/rime/fused/transformers/parangle.py#L29-L34

which I believe means that the beam implementation will also exhibit the same problems as enabling the parallactic angle rotation. I may, however, be completely mistaken.

Yes, if there's an issue affecting feed rotation, then it will affect the beam because they both depend on transformation of parallactic angles

[JSKenyon]

I can confirm that the GC is not the culprit. I have written a tiny hack for casa_parallactic_angles which spins off a process to run the measures server each time the function gets called. This seems to resolve the problem.

Edit: Though how robust it is I do not yet know. But very cool that it works from an objmode call inside the fused RIME. :-)

[sjperkins]

I can confirm that the GC is not the culprit. I have written a tiny hack for casa_parallactic_angles which spins off a process to run the measures server each time the function gets called. This seems to resolve the problem.

Edit: Though how robust it is I do not yet know. But very cool that it works from an objmode call inside the fused RIME. :-)

Could you please push your changes up to a branch, in whatever exploratory state it's in?

[JSKenyon]

Will do!

[JSKenyon]

My hack currently lives in https://github.com/ska-sa/codex-africanus/tree/improve_parangles.

[JSKenyon]

This definitely doesn't seem to work with multiple workers yet. Does work from threads. So there will be some pickling required somewhere.