Halting tasks on web UI does not terminate task in celery, and general HPC questions

[hoangthienan95]

Hi there, thank you for the awesome project, I just discovered it and I think it's perfect for distributing scripts with intuitive in a research lab setting for wet lab colleagues to use. I'm new so I'm encountering alot of problems and have searched resources without success.

I'm running the app locally, running python manage.py runserver on one terminal and celery -A WooeyApp worker -c 1 --beat -l info on a separate one. Using all the default settings and configurations.

1. One of the problems is that when I press the Stop button on the web ui of a script, the UI says that the status of the job is halted, yet if I wait for some time and refresh the page, the terminal output updates. My script does a loop and even after halting it printed the output of the next 10 loops. Any tasks I submit thereafter cannot run. On the terminal running celery I get (from when I click Stop):
   

What's going on here? Where should I look for problems? I'm very new to django and celery too.

2. Are there an example of wooey on hpc systems? I'm trying to distribute a script to members of my lab, who also share the same file system that's connected to a compute cluster. The script I'm distributing sends multiple jobs to the LSF HPC scheduler. I want them to use Wooey's GUI to give all the input parameters (mostly file paths on the shared file systems) but the script will launch jobs under their own usernames on LSF and not mine. Is there a way to accomplish this? I published a test version in the local network and right now whoever submits a job on the Wooey app will trigger computation on my side (terminal running celery picks up the job), or so I thought. I guess I'm confused on where computation is happening and how we can change that.

[Chris7]

Hi @hoangthienan95,

With respect to 1) can you give some information on your OS? Are you running windows, and if so, are you using the eventlet pool (documented here: https://wooey.readthedocs.io/en/latest/running_wooey.html)

I can respond in more detail to 2 later.

[hoangthienan95]

Hi @Chris7 I'm running Wooey on a remote machine (a login node on the LSF cluster I mentioned). The specs for this machine is below, it's Linux and not Windows. I used the command celery -A your_project_name worker -c 1 --beat -l info to start the Celery process.

!uname -a
>Linux tak 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

!lshw -class cpu
WARNING: you should run this program as super-user.
  *-cpu:0
       product: Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz
       vendor: Intel Corp.
       physical id: 1
       bus info: cpu@0
       size: 3113MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat cpufreq
  *-cpu:1
       product: Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz
       vendor: Intel Corp.
       physical id: 2
       bus info: cpu@1
       size: 2839MHz
       capacity: 2933MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat cpufreq
WARNING: output may be incomplete or inaccurate, you should run this program as super-user.

I also get errors from the first moment I start celery process like mentioned in #261, but they seemed harmless (?) . Will recreate the error tomorrow.

[Chris7]

Thanks -- what broker are you using? The no-configuration broker is really not suited for anything advanced (it's a file transport layer)

[hoangthienan95]

I didn't change anything from the defaults. Just did wooify -p ProjectNameto start a project from scratch and then celery -A your_project_name worker -c 1 --beat -l info and python manage.py runserver. Where do I go to learn more about brokers and how to connect them to Wooey?

I'd be more than willing to write a detailed tutorial on Wooey for a less technical audience like me, who would want to set it up and allow it to work out of the box, if such thing has not been done before. I read a bunch of tutorials on celery, but the scope of them is much larger than what is concerned with Wooey.

I'm using Wooey managed through Anaconda:

wooey                     0.10.1                   pypi_0    pypi

[Chris7]

That would be great -- to get setup with a different broker, you would setup a rabbitmq server (https://www.rabbitmq.com/) and in set the CELERY_BROKER_URL to its address (this is found in your settings/user_settings.py). The environment variable AMQP_URL can be used to set that without altering any code as well. There are some free hosted solutions for AMQP like https://www.cloudamqp.com/plans.html, but you would have to change a few things to avoid hitting the message limits per month (wooey.herokuapp.com uses the free tier of cloudamqp)

[razermouse]

Hey Chris, would there be any interest in a free Amazon AMI of Wooey that's already completely running?

[Chris7]

I would love an AMI of Wooey. My only preference would be it's made somehow with infrastructure as code like packer so it's easy to update and everyone knows what's running.

[andchir]

0.13.2rc1 - RabbitMQ - works fine, but options "broker_url, broker_transport_options" in settings don't work now.
broker_url = "filesystem://" - does not work (Worked in previous release).