CASA concat MS error

[Victoria-Samboco]

I wrote a Stimela recipe step to concatenate scans using the task concat from CASA, but I'm having a SyntaxError, which I need help understanding how to fix; please advise ...

───────────────────────────────────────── boom.concat_scans ──────────────────────────────────────────
2023-05-03 11:44:04 STIMELA.boom.concat_scans INFO: ### validated inputs
2023-05-03 11:44:04 STIMELA.boom.concat_scans INFO: cab casa:
2023-05-03 11:44:04 STIMELA.boom.concat_scans INFO:   vis_list = 
['obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_0.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_1.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_10.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_11.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_12.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_15.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_16.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_17.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_18.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_19.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_2.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_20.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_21.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_22.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_3.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_4.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_5.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_6.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_7.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_8.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_9.ms']
2023-05-03 11:44:04 STIMELA.boom.concat_scans INFO:   timesort = True
2023-05-03 11:44:04 STIMELA.boom.concat_scans INFO:   concatvis = 
obs3/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms
### running casa --log2term --nologger --nologfile -c import sys, json; kw=json.loads(sys.argv[-1]); 
kw={key: (str(value) if type(value) is unicode else value) for key, value in kw. items()}; casa 
--log2term --nologger --nologfile -c 
"concat(vis='['obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_0.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_1.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_10.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_11.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_12.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_15.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_16.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_17.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_18.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_19.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_2.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_20.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_21.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_22.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_3.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_4.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_5.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_6.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_7.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_8.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_9.ms']', 
concatvis='obs3/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms', timesort=True)"(**kw); 
{"vis_list": ["obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_0.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_1.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_10.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_11.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_12.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_15.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_16.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_17.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_18.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_19.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_2.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_20.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_21.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_22.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_3.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_4.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_5.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_6.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_7.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_8.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_9.ms"], "time sort": true, 
"concatvis": "obs3/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms"}
# 
# =========================================
# The start-up time of CASA may vary
# depending on whether the shared libraries
# are cached or not.
# =========================================
# 
# /opt/causality/lib/python2.7/site-packages/IPython/core/interactive shell.py:724: UserWarning: 
Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the 
virtualenv.
#   warn("Attempting to work in a virtualenv. If you encounter problems, please "
# IPython 5.1.0 -- An enhanced Interactive Python.
# 
# CASA 5.6.0-60   -- Common Astronomy Software Applications
# 
# Found an existing telemetry logfile: 
/home/samboco/.casa/casastats-560-60-1022843ce4242e0312-20230427-162428.log
# Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO.
# You can disable telemetry by adding the following line to your ~/.casarc file:
# EnableTelemetry: False
# 2023-05-03 09:44:08   INFO    ::casa  CASA Version  5.6.0-60
# --> CrashReporter initialized.
# Traceback (most recent call last):
#   File "/opt/casalite/lib/python2.7/init_welcome.py", line 60, in <module>
#     exec(casa['flags'].execute[0])
#   File "<string>", line 1
#     import sys, json; kw=json.loads(sys.argv[-1]); kw={key: (str(value) if type(value) is unicode 
else value) for key, value in kw.items()}; casa --log2term --nologger --nologfile -c 
"concat(vis='['obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_0.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_1.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_10.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_11.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_12.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_15.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_16.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_17.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_18.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_19.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_2.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_20.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_21.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_22.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_3.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_4.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_5.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_6.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_7.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_8.ms', 
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_9.ms']', 
concatvis='obs3/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms', timesort=True)"(**kw);
#                                                                                                     
^
# SyntaxError: invalid syntax

Here is the cab

  concat_MS:
    command:  casa --log2term --nologger --nologfile -c "concat(vis='{current.vis_list}', concatvis='{current.concatvis}', timesort=True)"
    flavour: casa-task
    inputs:
      vis_list:
        dtype: List[str]
        required: true
      commands:
        dtype: List[str]
        policies:
          repeat: list
      timesort:
        dtype: bool  
      concatvis:
        dtype: str
        #default: '{recipe.ms}-output'
        required: true

and recipe step

    concat_scans:
      cab: concat_MS
      info: ' Concatenate multiple measurement sets into a single output.'
      params:
        timesort: true
        vis_list: =GLOB("{recipe.dir_out}/{recipe.perscan_dir_out}/*scan*")
        concatvis: ="{recipe.dir_out}" + "/" + STRIPEXT(BASENAME(recipe.ms)) + "_concat.ms"

[o-smirnov]

Firstly, these issues belong on the "new" stimela repo, https://github.com/caracal-pipeline/stimela

Secondly, if you use the casa-task flavour, the inputs should not be included in the command, stimela will form up the command automatically. So rather try:

 concat_MS:
    command:  concat
    flavour: casa-task
    inputs:
      vis:
        dtype: List[MS]
        required: true
      timesort:
        dtype: bool  
    outputs:
      concatvis:
        dtype: MS
        required: true

[Victoria-Samboco]

Firstly, these issues belong on the "new" stimela repo, https://github.com/caracal-pipeline/stimela

Noted. Now should I reply here or create the same issue in the new repo?

Secondly, if you use the casa-task flavour, the inputs should not be included in the command, stimela will form up the command automatically. So rather try:

 concat_MS:
    command:  concat
    flavour: casa-task
    inputs:
      vis:
        dtype: List[MS]
        required: true
      timesort:
        dtype: bool  
    outputs:
      concatvis:
        dtype: MS
        required: true

Additonaly I've run it this way and it returned a different error

# /opt/casalite/lib/python2.7/site-packages/IPython/core/interactiveshell.py:724: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the 
virtualenv.
#   warn("Attempting to work in a virtualenv. If you encounter problems, please "
# IPython 5.1.0 -- An enhanced Interactive Python.
# 
# CASA 5.6.0-60   -- Common Astronomy Software Applications
# 
# Found an existing telemetry logfile: /home/samboco/.casa/casastats-560-60-1022843ce4242e0312-20230427-162428.log
# Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO.
# You can disable telemetry by adding the following line to your ~/.casarc file:
# EnableTelemetry: False
# 2023-05-03 19:57:09   INFO    ::casa  CASA Version  5.6.0-60
# --> CrashReporter initialized.
# 2023-05-03 19:57:11   INFO    concat::utils::verify   Verifying arguments....File existence checking failed for vis []
# 2023-05-03 19:57:11   WARN    concat::utils::verify   Argument vis failed to verify.
⚠  2023-05-03 19:57:11  SEVERE  concat::::      An error occurred running task concat.
2023-05-03 21:57:12 STIMELA.boom.concat_scans ERROR: declaring 'concat' as failed based on its output
2023-05-03 21:57:12 STIMELA.boom.concat_scans INFO: concat returns exit code 0 after 0:00:06
2023-05-03 21:57:12 STIMELA.boom.concat_scans ERROR: step 'boom.concat_scans' has failed, aborting the recipe
──────────────────────────────────────────────────────────────────────────────────── detailed error report follows ─────────────────────────────────────────────────────────────────────────────────────
        ⚠ step 'boom.concat_scans' has failed, aborting the recipe
        └── error running cab 'concat'
            └── CASA error:     SEVERE  concat::::      An error occurred running task concat.

The directory to the vis is correct and it returns a list of MS but the concat command fails to validate it. I was thinking here that it might be the parameter type. In the documentation says that the vis parameter is an stringArray type.

I didn't find this key type in Stimela documentation. So if this is the case how we can deal with it?

[o-smirnov]

Show the log higher up please? I want to see what stimela says before this happens.

[o-smirnov]

It says

File existence checking failed for vis []

...so are you sure you're not passing it an empty list of MSs? Please double-check.

[Victoria-Samboco]

Show the log higher up please? I want to see what stimela says before this happens.

Here is the full log

───────────────────────────────────────────── boom.concat_scans ─────────────────────────────────────────────
2023-05-04 01:15:47 STIMELA.boom.concat_scans INFO: ### validated inputs
2023-05-04 01:15:47 STIMELA.boom.concat_scans INFO: cab concat:
2023-05-04 01:15:47 STIMELA.boom.concat_scans INFO:   vis =
['obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_0.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_1.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_10.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_11.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_12.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_15.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_16.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_17.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_18.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_19.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_2.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_20.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_21.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_22.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_3.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_4.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_5.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_6.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_7.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_8.ms',
'obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_9.ms']
2023-05-04 01:15:47 STIMELA.boom.concat_scans INFO:   timesort = True
2023-05-04 01:15:47 STIMELA.boom.concat_scans INFO:   concatvis =
obs3/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms
### running casa --log2term --nologger --nologfile -c import sys, json; kw=json.loads(sys.argv[-1]); kw={key:
(str(value) if type(value) is unicode else value) for key, value in kw.items()}; concat(**kw); {"vis": 
["obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_0.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_1.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_10.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_11.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_12.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_15.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_16.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_17.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_18.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_19.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_2.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_20.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_21.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_22.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_3.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_4.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_5.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_6.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_7.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_8.ms", 
"obs3/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_9.ms"], "timesort": true, 
"concatvis": "obs3/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms"}
# 
# =========================================
# The start-up time of CASA may vary
# depending on whether the shared libraries
# are cached or not.
# =========================================
# 
# /opt/casalite/lib/python2.7/site-packages/IPython/core/interactiveshell.py:724: UserWarning: Attempting to
work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
#   warn("Attempting to work in a virtualenv. If you encounter problems, please "
# IPython 5.1.0 -- An enhanced Interactive Python.
# 
# CASA 5.6.0-60   -- Common Astronomy Software Applications
# 
# Found an existing telemetry logfile: /home/samboco/.casa/casastats-560-60-1022843ce4242e0312-20230427-162428.log
# Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO.
# You can disable telemetry by adding the following line to your ~/.casarc file:
# EnableTelemetry: False
# 2023-05-03 23:16:30   INFO    ::casa  CASA Version  5.6.0-60
# --> CrashReporter initialized.
# 2023-05-03 23:16:55   INFO    concat::utils::verify   Verifying arguments....File existence checking failed for vis []
# 2023-05-03 23:16:55   WARN    concat::utils::verify   Argument vis failed to verify.
⚠  2023-05-03 23:16:55  SEVERE  concat::::      An error occurred running task concat.
2023-05-04 01:16:56 STIMELA.boom.concat_scans ERROR: declaring 'concat' as failed based on its output
2023-05-04 01:16:56 STIMELA.boom.concat_scans INFO: concat returns exit code 0 after 0:01:09
2023-05-04 01:16:56 STIMELA.boom.concat_scans ERROR: step 'boom.concat_scans' has failed, aborting the recipe
─────────────────────────────────────────────── detailed error report follows ───────────────────────────────────────────────
        ⚠ step 'boom.concat_scans' has failed, aborting the recipe
        └── error running cab 'concat'
            └── CASA error:     SEVERE  concat::::      An error occurred running task concat.

[o-smirnov]

OK let me try to run it as you. Which machine, directory, virtual environment, stimela command?

[Victoria-Samboco]

OK let me try to run it as you. Which machine, directory, virtual environment, stimela command?

Garfunkel,
directory: /home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/

Virtual environment: /home/samboco/Virtual_enviroments/stimela_env

Stimela command
stimela run solarKAT.yml boom obs=UHF -s concat_scans

[o-smirnov]

I get this:

2023-05-04 23:20:08 STIMELA.boom.concat_scans ERROR: error validating inputs:                                                                                                                                     
───────────────────────────────────────────────────────────────────────────────────────── detailed error report follows ──────────────────────────────────────────────────────────────────────────────────────────
        ⚠ error validating inputs:                                                                                                                                                                                
        └── 'boom.concat_scans.vis': file(s) don't exist                                                                                                                                                          
2023-05-04 23:20:08 STIMELA.boom.concat_scans ERROR: ### summary of inputs follows                                                                                                                                
2023-05-04 23:20:08 STIMELA.boom.concat_scans ERROR: cab concat:                                                                                                                                                  
2023-05-04 23:20:08 STIMELA.boom.concat_scans ERROR:   vis = []                                                                                                                                                   
2023-05-04 23:20:08 STIMELA.boom.concat_scans ERROR:   timesort = True                                                                                                                                            
2023-05-04 23:20:08 STIMELA.boom.concat_scans ERROR:   concatvis = obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms                                                                         
2023-05-04 23:20:08 STIMELA ERROR: run failed after 0:00:00                                                                                                                                                       
───────────────────────────────────────────────────────────────────────────────────────── detailed error report follows ──────────────────────────────────────────────────────────────────────────────────────────
        ⚠ run failed after 0:00:00                                                                                                                                                                                
        └── step 'boom.concat_scans' has failed, aborting the recipe                                                                                                                                              
            └── 'boom.concat_scans.vis': file(s) don't exist                                    

Am I missing a parameter to the recipe?

[Victoria-Samboco]

Oh, sorry, the files were not there then, but it is fixed now. Please try again...

[o-smirnov]

Ah ok, I see the problem now. Same as caracal-pipeline/stimela#123, which is fixed. Update stimela to master and try again.

[Victoria-Samboco]

Ah ok, I see the problem now. Same as caracal-pipeline/stimela#123, which is fixed. Update stimela to master and try again.

Did that and then I had a different error

# /opt/casalite/lib/python2.7/site-packages/IPython/core/interactiveshell.py:724: UserWarning: Attempting to work
in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
#   warn("Attempting to work in a virtualenv. If you encounter problems, please "
# IPython 5.1.0 -- An enhanced Interactive Python.
# 
# CASA 5.6.0-60   -- Common Astronomy Software Applications
# 
# Found an existing telemetry logfile: 
/home/samboco/.casa/casastats-560-60-1022843ce4242e0312-20230505-064409.log
# Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO.
# You can disable telemetry by adding the following line to your ~/.casarc file:
# EnableTelemetry: False
# 2023-05-08 08:25:27   INFO    ::casa  CASA Version  5.6.0-60
# --> CrashReporter initialized.
# Traceback (most recent call last):
#   File "/opt/casalite/lib/python2.7/init_welcome.py", line 60, in <module>
#     exec(casa['flags'].execute[0])
#   File "<string>", line 1, in <module>
# ImportError: No module named yaml
2023-05-08 10:25:46 STIMELA.boom.concat_scans ERROR: step 'boom.concat_scans' has failed, aborting the recipe
───────────────────────────────────────── detailed error report follows ─────────────────────────────────────────
        ⚠ step 'boom.concat_scans' has failed, aborting the recipe
        └── error running cab 'concat'
            └── concat returns error code 1 after 0:00:56

I succeeded to fix the error by installing the module pyyaml for python2.7 which CASA is based on.

[o-smirnov]

I succeeded to fix the error by installing the module pyyaml for python2.7 which CASA is based on.

Well done, though I'm a bit surprised this was necessary -- which casa are you running?

[Victoria-Samboco]

I succeeded to fix the error by installing the module pyyaml for python2.7 which CASA is based on.

Well done, though I'm a bit surprised this was necessary -- which casa are you running?

I'm running CASA Version 5.6.0-60

[Victoria-Samboco]

well, it runs but then crashes in the 14th scan for some reason...

....
# 2023-05-08 09:35:23   INFO    concat::::      concatenating
obs3/ms_scans/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_13.ms into
obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms
# 2023-05-08 09:35:23   INFO    MSConcat::concatenate   Appending
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/ms_scans/1583662427_sdp_l0.1024ch-J033230_280757-
corr_self_scan_13.ms to
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-cor
r_self_concat.ms
# 2023-05-08 09:35:23   INFO    MSConcat::concatenate   Added 0 rows and matched 4 from the state
subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate   Added 1 rows to the source subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate   Added 0 rows and matched 1 from the data
description subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate   Added 0 rows and matched 1 from the spectral
window subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate   Removed 1 redundant rows from the source
subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate   Added 0 rows and matched 57 from the antenna
subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate + Added 0 rows to the feed subtable
# 2023-05-08 09:35:24   INFO    MSConcat::concatenate + Added 0 rows and matched 1 from the field
subtable
# 2023-05-08 09:35:24   INFO    MSConcat::copyObservation       Added 0 rows and matched 1 rows in the
observation subtable.
# 2023-05-08 09:35:24   INFO    MSConcat::copyPointing  No valid pointing tables present. Result won't
have one either.
# 2023-05-08 09:35:24   INFO    MSConcat::copySysCal    No valid syscal tables present. Result won't
have one either.
# 2023-05-08 09:35:24   INFO    MSConcat::copyWeather   No valid weather tables present. Result won't
have one either.
# 2023-05-08 09:38:31   INFO    concat::::      concatenating
obs3/ms_scans/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_scan_14.ms into
obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-corr_self_concat.ms
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Appending
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/ms_scans/1583662427_sdp_l0.1024ch-J033230_280757-
corr_self_scan_14.ms to
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-cor
r_self_concat.ms
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Added 0 rows and matched 4 from the state
subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Added 1 rows to the source subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Added 0 rows and matched 1 from the data
description subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Added 0 rows and matched 1 from the spectral
window subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Removed 1 redundant rows from the source
subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate   Added 0 rows and matched 57 from the antenna
subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate + Added 0 rows to the feed subtable
# 2023-05-08 09:38:31   INFO    MSConcat::concatenate + Added 0 rows and matched 1 from the field
subtable
# 2023-05-08 09:38:31   INFO    MSConcat::copyObservation       Added 0 rows and matched 1 rows in the
observation subtable.
# 2023-05-08 09:38:31   INFO    MSConcat::copyPointing  No valid pointing tables present. Result won't
have one either.
# 2023-05-08 09:38:31   INFO    MSConcat::copySysCal    No valid syscal tables present. Result won't
have one either.
# 2023-05-08 09:38:31   INFO    MSConcat::copyWeather   No valid weather tables present. Result won't
have one either.
2023-05-08 09:40:36  SEVERE  ms::concatenate Exception Reported: FiledesIO: write error in 
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-cor
r_self_concat.ms/table.f17_TSM1: Success
⚠  *** Error *** FiledesIO: write error in 
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-cor
r_self_concat.ms/table.f17_TSM1: Success
⚠  2023-05-08 09:40:36  SEVERE  concat::::      An error occurred running task concat.
# terminate called after throwing an instance of 'casacore::AipsError'
#   what():  FilebufIO: write error for file
/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J033230_280757-cor
r_self_concat.ms/table.dat: Disk quota exceeded
# No LSB modules are available.
#
# --------------------------------------------------------------
# CASA has crashed...
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
# A crash report is being generated and submitted. The report
# will contain information about CASA's environment when the
# crash occurred (e.g. CASA log, call stack, CPU information).
# This should not take too long...
# --------------------------------------------------------------
#
2023-05-08 11:40:48 STIMELA.boom.concat_scans ERROR: declaring 'concat' as failed based on its output
2023-05-08 11:40:48 STIMELA.boom.concat_scans ERROR: step 'boom.concat_scans' has failed, aborting the 
recipe
──────────────────────────────────── detailed error report follows ─────────────────────────────────────
        ⚠ step 'boom.concat_scans' has failed, aborting the recipe
        └── error running cab 'concat'
            ├── CASA error:     SEVERE  ms::concatenate Exception Reported: FiledesIO: write error in
            │   /home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J0
            │   33230_280757-corr_self_concat.ms/table.f17_TSM1: Success
            ├── CASA error: *** Error *** FiledesIO: write error in
            │   /home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/obs3/msdir/1583662427_sdp_l0.1024ch-J0
            │   33230_280757-corr_self_concat.ms/table.f17_TSM1: Success
            ├── CASA error:     SEVERE  concat::::      An error occurred running task concat.
            └── concat returns error code -6 after 0:42:52
────────────────────────────────────────── profiling results ───────────────────────────────────────────

[o-smirnov]

What does which casa say?