Adnane: R jobs on ShARC are killed due to exceeding requested memory limit

[willfurnass]

@Adnane017 reported that when he tries to run a job containing an R script on ShARC it is killed by the job scheduler due the job exceeding the requested memory limit of 64GB:

$ qacct -u ld1ae -d 10 -j 1424695 | grep -E '^(category|failed|maxvmem|jobnumber)'
jobnumber    1424695             
failed       37  : qmaster enforced h_rt, h_cpu, or h_vmem limit
maxvmem      64.004GB
category     -U odk,rse -u ld1ae -l h_vmem=64G -I y -P SHEFFIELD

I suggested increasing the memory limit to 192GB and running some profiling but that was still insufficient RAM:

$ qacct -u ld1ae -d 10 -j 1428046 | grep -E '^(category|failed|maxvmem|jobnumber)'
jobnumber    1428046             
failed       37  : qmaster enforced h_rt, h_cpu, or h_vmem limit
maxvmem      192.305GB
category     -U odk,rse -u ld1ae -q rse-interactive.q -l h_vmem=192G -I y -P rse

[willfurnass]

I then asked @Adnane017 to run some memory profiling so we could see how usage increased during the execution of his program:

# Start profiling, saving the output to a file
Rprof(filename = "Rprof.out", gc.profiling = TRUE, line.profiling = TRUE, memory.profiling = TRUE)

# The bulk of code (excluding the loading of libraries)

# Stop profiling
Rprof(NULL)

We should then be able to analyse the resulting RProf.out on a desktop machine using e.g. profvis.

An initial look at the code suggests that at some point a 53000x53000 matrix is formed, which should require about 21GB RAM if it contains double-precision floating point numbers; we need to look closely at the code and profiling results to determine the cause of the issue.

[willfurnass]

@Adnane017 Is your code in a repository within the ooominds GitHub organisation?

[Adnane017]

Here is a simplified version of the code with the function "estimateActivations", which causes the memory problem. estimateActivations is part of the ndl package, so you will need to install it as well. When I tried to run the code on my machine, R complained that it cannot allocate vector of size 10.3 Gb (I have 16Gb of RAM). Best Adnane

…

On 28 June 2018 at 15:56, Will Furnass ***@***.***> wrote: @Adnane017 <https://github.com/Adnane017> Is your code in a repository within the ooominds GitHub repository <https://github.com/ooominds>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AeXuQ7qPziqnsJPGyb_Ak5Qbjy1IDMmCks5uBO63gaJpZM4U7iIL> .

[willfurnass]

@Adnane017: you replied to an email sent by GitHub; I can see your comments but not your attachment.

Is your code in a GitHub repository? If not, can we create a GitHub repository for it? It will make it easier for us to collaborate on your code.

[Adnane017]

This matter is no longer urgent (we are focusing on some other projects for now), so maybe we should leave this for later. But thanks a lot for your help.

…

On 6 July 2018 at 10:00, Will Furnass ***@***.***> wrote: @Adnane017 <https://github.com/Adnane017>: you replied to an email sent by GitHub; I can see your comments but not your attachment. Is your code in a GitHub repository? If not, can we create a GitHub repository for it? It will make it easier for us to collaborate on your code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AeXuQ5-LjAjtqfvnE_CZvAW7jwwID-Opks5uDycXgaJpZM4U7iIL> .