# FATAL: while parsing bind path: wrong bind syntax

[viralp]

Hi

I am getting the following singularity error in stimela.
log-A168_D_Config_reduction_script.txt
log-A168_D_Config_reduction_script-imaging_round_init.txt

2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: ---------------------------------
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Stimela version 1.7.6
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Running: A168 D Config reduction script
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: ---------------------------------
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Config directory cannot be found. Will create /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/stimela_parameter_files
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Adding cab '/export/home/viralp/singularity_images/stimela_wsclean_1.7.4.sif' (3.1) to recipe, container name 'imaging_round_init'
2023-04-05 15:58:41 STIMELA.imaging_round_init INFO: job started at 2023-04-05 15:58:41.343580
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: STEP 1 :: imaging_round_init:: Make initial image
2023-04-05 15:58:41 STIMELA INFO: Validating parameters for cab wsclean (stimela/wsclean:1.7.4)
2023-04-05 15:58:41 STIMELA INFO: Parameters validated and saved to /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/stimela_parameter_files/imaging_round_init-14042593832337616807319211861491.json
2023-04-05 15:58:41 STIMELA.imaging_round_init INFO: Starting container [imaging_round_init-14042593832337616807319211861491]. Timeout set to -1. The container ID is printed below.
# running cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall  --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/stimela_parameter_files/imaging_round_init-14042593832337616807319211861491.json:/stimela_mount/configfile:ro --bind /users/vparekh/venv3/lib/python3.8/site-packages/stimela/cargo/cab/wsclean/src:/stimela_mount/code:ro --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/passwd:/etc/passwd:rw --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/group:/etc/group:rw --bind /users/vparekh/venv3/bin/stimela_runscript:/singularity:ro --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/msdir:/stimela_mount/msdir:rw --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/input:/stimela_mount/input:ro --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/output_A168_relic_target:/stimela_mount/output:rw --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/output_A168_relic_target/tmp:/stimela_mount/output/tmp:rw /export/home/viralp/singularity_images/stimela_wsclean_1.7.4.sif /singularity
# FATAL:   while parsing bind path: wrong bind syntax: /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/passwd:/etc/passwd:rw
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall returns error code 255
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: job failed at 2023-04-05 15:58:41.426185 after 0:00:00.082605
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: Traceback (most recent call last):
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     job.run_job()
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: stimela.utils.StimelaCabRuntimeError: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall returns error code 255
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Completed jobs : []
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Remaining jobs : []
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Saving pipeline information in .last_A168_D_Config_reduction_script.json
Traceback (most recent call last):
  File "/users/vparekh/venv3/bin/stimela", line 11, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 409, in main
    _cmd(argv)
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 199, in run
    exec(stdr.read(), _globals)
  File "<string>", line 69, in <module>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run
    raise PipelineException(exc, self.completed, job, self.remaining) from None
stimela.exceptions.PipelineException: Job 'imaging_round_init:: Make initial image' failed: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall returns error code 255

stimela_script_vla.txt

[o-smirnov]

This looks to be the real error, singularity doesn't understand the command line Stimela forms up:

while parsing bind path: wrong bind syntax:

@viralp what version of singularity is on this box? I presume it's either too old (not our problem then) or too new (in which case it is...)

[viralp]

Seems to be old version.

singularity version 3.6.1+3-gb94fa3544

[viralp]

Ok singularity has been updated to 3.11. Now getting the following error

#  == Constructing PSF ==
# Detected 62.4 GB of system memory, usage not limited.
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/A168_VLA_L_D_targets.ms
# Determining min and max w & theoretical beam size... Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 69, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['wsclean', '-name', '/stimela_mount/output/17A-187.sb33657704.eb33666723.57871.671216585644init', '-j', '32', '-mem', '100', '-weight', 'briggs', '-0.5', '-size', '6000', '6000', '-scale', '0.06arcsec', '-channels-out', '32', '-no-small-inversion', '-pol', 'I', '-data-column', 'DATA', '-niter', '2000000', '-auto-threshold', '0.5', '-auto-mask', '5.0', '-mgain', '0.9', '-join-channels', '-fit-spectral-pol', '4', '-nmiter', '20', '/stimela_mount/msdir/A168_VLA_L_D_targets.ms']' died with <Signals.SIGKILL: 9>.
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: job failed at 2023-04-18 14:30:08.140287 after 0:00:32.468797
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: Traceback (most recent call last):
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     job.run_job()
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: stimela.utils.StimelaCabRuntimeError: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1
2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Completed jobs : []
2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Remaining jobs : []
2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Saving pipeline information in .last_A168_Config_reduction_script__Sequential_.json
Traceback (most recent call last):
  File "/users/vparekh/venv3/bin/stimela", line 11, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 409, in main
    _cmd(argv)
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 199, in run
    exec(stdr.read(), _globals)
  File "<string>", line 76, in <module>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run
    raise PipelineException(exc, self.completed, job, self.remaining) from None
stimela.exceptions.PipelineException: Job 'imaging_round_init:: Make initial image' failed: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1

[bennahugo]

Don't think this has anything to do with singularity - it definitely runs and logs. Most likely the out of memory killer. Contact your system admin to adjust cgroups settings for your machine if there are limits set.

…

On Tue, 18 Apr 2023, 22:45 Viral Parekh, ***@***.***> wrote: Ok singularity has been updated to 3.11. Now getting the following error # == Constructing PSF == # Detected 62.4 GB of system memory, usage not limited. # Opening reordered part 0 spw 0 for /stimela_mount/msdir/A168_VLA_L_D_targets.ms # Determining min and max w & theoretical beam size... Traceback (most recent call last): # File "/stimela_mount/code/run.py", line 69, in <module> # subprocess.check_call(shlex.split(_runc)) # File "/usr/lib/python3.8/subprocess.py", line 364, in check_call # raise CalledProcessError(retcode, cmd) # subprocess.CalledProcessError: Command '['wsclean', '-name', '/stimela_mount/output/17A-187.sb33657704.eb33666723.57871.671216585644init', '-j', '32', '-mem', '100', '-weight', 'briggs', '-0.5', '-size', '6000', '6000', '-scale', '0.06arcsec', '-channels-out', '32', '-no-small-inversion', '-pol', 'I', '-data-column', 'DATA', '-niter', '2000000', '-auto-threshold', '0.5', '-auto-mask', '5.0', '-mgain', '0.9', '-join-channels', '-fit-spectral-pol', '4', '-nmiter', '20', '/stimela_mount/msdir/A168_VLA_L_D_targets.ms']' died with <Signals.SIGKILL: 9>. 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: job failed at 2023-04-18 14:30:08.140287 after 0:00:32.468797 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: Traceback (most recent call last): 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: job.run_job() 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: self.job.run(output_wrangler=self.apply_output_wranglers) 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall", 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: stimela.utils.StimelaCabRuntimeError: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1 2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Completed jobs : [] 2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Remaining jobs : [] 2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Saving pipeline information in .last_A168_Config_reduction_script__Sequential_.json Traceback (most recent call last): File "/users/vparekh/venv3/bin/stimela", line 11, in <module> main.main([a for a in sys.argv[1:]]) File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 409, in main _cmd(argv) File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 199, in run exec(stdr.read(), _globals) File "<string>", line 76, in <module> File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run raise PipelineException(exc, self.completed, job, self.remaining) from None stimela.exceptions.PipelineException: Job 'imaging_round_init:: Make initial image' failed: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1 — Reply to this email directly, view it on GitHub <#776 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6RVO5J2M32GVXCTPGTXB34LLANCNFSM6AAAAAAWUUFOMM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>