Skip to content

problem with running example parallel futures using batchtools_lsf #76

Open
@haibol2016

Description

@haibol2016

I am trying to run the example code in the future.batchtools Git Hub README, as follows.
But I failed due to error message shown below.

Could you help me out?
Thank you.

Haibo

library(future.batchtools)

plan(batchtools_lsf(template = "lsf.tmpl", workers = 2,
                    resources = list(walltime = 3600*8, num.cpu = 4,
                                        mpp = 1024*4, queue = "long")))

x %<-% { Sys.sleep(5); 3.14 }
y %<-% { Sys.sleep(5); 2.71 }
x + y

I got the following error message inconsistently

  1. err message 1:

Error: Log file '/project/umw_arthur_mercurio/Haibo/InPAS/.future/20211220_104348-sqIN0l/batchtools_2107070426/logs/job74e5f3ce70dd44fe91a775b79b221d03.log'
for job with id 1 not available

  1. err message 2:
    Error: BatchtoolsExpiration: Future ('') expired (registry path /project/umw_arthur_mercurio/Haibo/InPAS/.future/20211220_104348-sqIN0l/batchtools_290284087)
    .. No logged output exist.

The contents of my lsf.tmpl file is as follow:

## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

#BSUB -n <%= resources$num.cpu %> # minmal numbers of processors required for a parallel job
#BSUB -R rusage[mem=<%= resources$mpp %>] # ask for memory per cpu
#BSUB -J <%= job.name %> # Name of the job
#BSUB -o <%= log.file %> # Output is sent to logfile, stdout + stderr by default
#BSUB -q <%= resources$queue %> # Job queue
#BSUB -W <%= round(resources$walltime / 60, 1) %> # Walltime (LSF requires minutes, batchtools uses seconds)
##BSUB -M <%= resources$memory %> # Memory requirements, e.g. "5000KB", "500MB", "5GB" etc.
#BSUB -R "span[hosts=1]" # All hosts on the same chassis"

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

>  sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.4 (Ootpa)

Matrix products: default
BLAS/LAPACK: /project/umw_mccb/bin/miniconda2/envs/seurat4/lib/libopenblasp-r0.3.15.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future.batchtools_0.10.0 future_1.23.0           

loaded via a namespace (and not attached):
 [1] parallelly_1.29.0 magrittr_2.0.1    hms_1.1.1         progress_1.2.2   
 [5] rappdirs_0.3.3    R6_2.5.1          brew_1.0-6        rlang_0.4.12     
 [9] fansi_0.5.0       globals_0.14.0    tools_4.1.0       parallel_4.1.0   
[13] checkmate_2.0.0   data.table_1.14.2 utf8_1.2.2        withr_2.4.3      
[17] ellipsis_0.3.2    base64url_1.4     digest_0.6.29     tibble_3.1.6     
[21] lifecycle_1.0.1   crayon_1.4.2      fs_1.5.2          vctrs_0.3.8      
[25] batchtools_0.9.15 codetools_0.2-18  stringi_1.7.6     pillar_1.6.4     
[29] compiler_4.1.0    backports_1.4.1   prettyunits_1.1.1 listenv_0.8.0    
[33] pkgconfig_2.0.3  

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions