Output fails when running with duration higher than 2000

[LucianSheen]

If this issue is intended to be a bug report, please fill out the following:

Expected behavior
Using 0.9.3 image in docker, but whenever I try to run the simulation with command run.cmd -d 2000 or more, it fails.

here is an example:

2023-04-22 02:25:06 >> --------------------------------------------------------------
2023-04-22 02:25:06 >> Command exited with -9: python sibernetic_c302.py
2023-04-22 02:25:06 -duration 2200.0
2023-04-22 02:25:06 -dt 0.005
2023-04-22 02:25:06 -dtNrn 0.05
2023-04-22 02:25:06 -logstep 100
2023-04-22 02:25:06 -device=CPU
2023-04-22 02:25:06 -configuration worm_crawl_half_resolution
2023-04-22 02:25:06 -reference FW
2023-04-22 02:25:06 -c302params C2
2023-04-22 02:25:06 -datareader UpdatedSpreadsheetDataReader2
2023-04-22 02:25:06 -outDir simulations
2023-04-22 02:25:06 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-04-22 02:25:06
2023-04-22 02:25:06 Exiting as the last command failed

Actual behavior

Steps to reproduce the behavior

---

If this issue is intended to be a beginner issue, please add all of the following:

• Requirements are clearly specified
• Links to relevant "getting started" documentation
• Tagged as "beginner"
• Tagged with language skill needed to complete
• Level of difficulty:
  • Trivial = Changing one-line of code somewhere.
  • Standalone = Changes & Development do not affect other components of project. Tasks can be on a critical path, but probably should not be.
  • Simple = from 2 hours work to maybe a week’s work (Thinking + Coding)

[pgleeson]

Thanks for reporting @LucianSheen. I've not had problems running the docker file myself recently, but admittedly I've not run it for very long simulations. I'll try again locally and see if I can reproduce. Just to clarify, did you rebuild the image from the master branch or use the dockerhub version?

[LucianSheen]

Hello Padraig, Thanks for the email, and I built the image using github repo(tried the both master branch and 0.9.3 and they both showed me this issue) I'll try the dockerhub version and will let you know, too. Best, Lucian

…

On Fri, 28 Apr 2023 at 18:04, Padraig Gleeson ***@***.***> wrote: Thanks for reporting @LucianSheen <https://github.com/LucianSheen>. I've not had problems running the docker file myself recently, but admittedly I've not run it for very long simulations. I'll try again locally and see if I can reproduce. Just to clarify, did you rebuild the image from the master branch or use the dockerhub version? — Reply to this email directly, view it on GitHub <#332 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7G37LVXGCRW3U5SWONBBNTXDP2ANANCNFSM6AAAAAAXKYA7DI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pgleeson]

So I've been trying to test this out. The 5 second run of the Docker image worked fine on my Linux machine, but took 10 hours. It has 20 processers and 64GB ram and after it finished the swap memory had been used, so I suspect it took up quite a bit of memory while running.

A simulation of 0.5 seconds on my more modest laptop (Mac M2) failed with a similar error to you. I suspect then that it's running out of memory in the image and failing. What OS/specs are your machine?

Unfortunately I suspect it will require quite a powerful machine to run the full thing, unless Sibernetic (or the Python interface) can be optimised more...

[LucianSheen]

Hello Padraig, I was running the simulation in Windows 10 with my desktop, which has got amd 3900x, 32gb ram, gtx 1070. I was suspecting the same because I failed with the docker pulled image, although I haven’t tried in Linux, but thank you so much for the update!

…

On Fri, 5 May 2023 at 18:24, Padraig Gleeson ***@***.***> wrote: So I've been trying to test this out. The 5 second run of the Docker image worked fine on my Linux machine, but took 10 hours. It has 20 processers and 64GB ram and after it finished the swap memory had been used, so I suspect it took up quite a bit of memory while running. A simulation of 0.5 seconds on my more modest laptop (Mac M2) failed with a similar error to you. I suspect then that it's running out of memory in the image and failing. What OS/specs are your machine? Unfortunately I suspect it will require quite a powerful machine to run the full thing, unless Sibernetic (or the Python interface) can be optimised more... — Reply to this email directly, view it on GitHub <#332 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7G37LU6MEEE5VTKIZE26M3XEUZTBANCNFSM6AAAAAAXKYA7DI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[LucianSheen]

Hello Padraig! I have been also working with robot project as well, and it seems like wormpy.py in WormHost lacks of some codes. In worm_run.py , which will run the robot, it has got: wormpy.get_segment_angles(segment) but there's no such function get_segment_angles in wormpy.py I tried to add: def get_segment_angles(segment): return _wormpy.get_segment_angles(segment) get_segment_angles = _wormpy.get_segment_angles but it seems _wormpy.pyd doesn't have get_segment_angles either. Do I need to calculate the angle per segments? if so, it would be great to get help for how to do this. Best, Lucain

…

On Fri, 5 May 2023 at 19:13, Lucian Shin ***@***.***> wrote: Hello Padraig, I was running the simulation in Windows 10 with my desktop, which has got amd 3900x, 32gb ram, gtx 1070. I was suspecting the same because I failed with the docker pulled image, although I haven’t tried in Linux, but thank you so much for the update! On Fri, 5 May 2023 at 18:24, Padraig Gleeson ***@***.***> wrote: > So I've been trying to test this out. The 5 second run of the Docker > image worked fine on my Linux machine, but took 10 hours. It has 20 > processers and 64GB ram and after it finished the swap memory had been > used, so I suspect it took up quite a bit of memory while running. > > A simulation of 0.5 seconds on my more modest laptop (Mac M2) failed with > a similar error to you. I suspect then that it's running out of memory in > the image and failing. What OS/specs are your machine? > > Unfortunately I suspect it will require quite a powerful machine to run > the full thing, unless Sibernetic (or the Python interface) can be > optimised more... > > — > Reply to this email directly, view it on GitHub > <#332 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A7G37LU6MEEE5VTKIZE26M3XEUZTBANCNFSM6AAAAAAXKYA7DI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[pgleeson]

Hi @LucianSheen. I've not been involved with that code, but I see @portegys over there has helped you out with some answers.

BTW, there is a new Python visualisation/replay tool for the Boyle Berri Cohen model at https://github.com/OpenSourceBrain/CelegansNeuromechanicalGaitModulation.

See also https://github.com/openworm/CE_locomotion

Huh, this is the same issue I've been having too...

[samwagg]

Edit: I had originally claimed that not much CPU was being used, but I realized I was only looking at 1 process within the container so deleted that sentence.

This happens to me too. It seems to happen when plots are being created. I'm including some extra lines of logs above the actual error for more context:

>>   Sibernetic:  Python >>  > Current NEURON time: 4999.989999933098 ms
>>   Sibernetic:  Python >>  < Current NEURON time: 4999.9949999330975 ms
>>   Sibernetic:  Problem passing output of a_MVL24 to muscle in Sibernetic: 'hoc.HocObject' object has no attribute 'a_MVL24' Note: not an issue as no muscle MVL24 in the real C. elegans
>>  /home/ow/c302/c302/c302_utils.py:80: UserWarning: FixedFormatter should only be used together with FixedLocator
>>    ax.set_xticklabels(labels)
>>  pyNeuroML >>> 18:46:26 - INFO - Generating plot: Membrane potentials of 39 neuron(s) (FW C2)
>>  pyNeuroML >>> 18:46:27 - INFO - Saving image to /home/ow/sibernetic/simulations/C2_FW_2025-04-04_14-25-08/traces_neuron_FW_C2.png of plot: Membrane potentials of 39 neuron(s) (FW C2)
>>  pyNeuroML >>> 18:46:28 - INFO - Saved image to simulations/C2_FW_2025-04-04_14-25-08/traces_neuron_FW_C2.png of plot: Membrane potentials of 39 neuron(s) (FW C2)
>>  /home/ow/c302/c302/c302_utils.py:80: UserWarning: FixedFormatter should only be used together with FixedLocator
>>    ax.set_xticklabels(labels)
>>  pyNeuroML >>> 18:46:57 - INFO - Generating plot: Membrane potentials of 95 muscle(s) (FW C2)
>>  pyNeuroML >>> 18:47:17 - INFO - Saving image to /home/ow/sibernetic/simulations/C2_FW_2025-04-04_14-25-08/traces_muscles_FW_C2.png of plot: Membrane potentials of 95 muscle(s) (FW C2)
>>  pyNeuroML >>> 18:47:20 - INFO - Saved image to simulations/C2_FW_2025-04-04_14-25-08/traces_muscles_FW_C2.png of plot: Membrane potentials of 95 muscle(s) (FW C2)
>>  /home/ow/c302/c302/c302_utils.py:80: UserWarning: FixedFormatter should only be used together with FixedLocator
>>    ax.set_xticklabels(labels)
>>  pyNeuroML >>> 18:47:27 - INFO - Generating plot: [Ca2+] of 39 neurons (FW C2)
>> --------------------------------------------------------------
>> Command exited with -9: python sibernetic_c302.py
                -duration 5000.0
                -dt 0.005
                -dtNrn 0.05
                -logstep 100
                -device=CPU
                -configuration worm_crawl_half_resolution
                -reference FW
                -c302params C2
                -datareader UpdatedSpreadsheetDataReader2
                -outDir simulations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

And here's a screenshot of my system specs. It took 4.5 hours to run and used the CPU (it seems from other issues I've seen on here that perhaps gpu doesn't work in the docker container?).

[samwagg]

Ah, it definitely is running out of memory. Found the kill in syslog:

kernel: [78372.756970] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-4b4310e89d7213b228422d28f2e34a73d72ae5b63ee0562f8ff31ffd4569dfaf.scope,task=python,pid=105543,uid=0
kernel: [78372.757042] Out of memory: Killed process 105543 (python) total-vm:63470148kB, anon-rss:61763060kB, file-rss:304kB, shmem-rss:0kB, UID:0 pgtables:122064kB oom_score_adj:0

But as I said above, this is happening after the simulation finishes.

Ah, it definitely is running out of memory. Found the kill in syslog:

kernel: [78372.756970] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-4b4310e89d7213b228422d28f2e34a73d72ae5b63ee0562f8ff31ffd4569dfaf.scope,task=python,pid=105543,uid=0
kernel: [78372.757042] Out of memory: Killed process 105543 (python) total-vm:63470148kB, anon-rss:61763060kB, file-rss:304kB, shmem-rss:0kB, UID:0 pgtables:122064kB oom_score_adj:0

But as I said above, this is happening after the simulation finishes.

Yeah it seemed it failed exporting the output and the temp files while it was running are hard to access from host