Estimate run time, ram, cpu required as a slurm job

[redbybeing]

Hi,

I have a dataset of 95,000 cells and 6 cell types.

I want to include other methods in addition to default methods in liana_wrap(), including CellChat.
With just default methods, it took ~20 min to run liana_wrap() in RStudio Server.

I had worked with CellChat before with my dataset. I don't remember the details but it took several hours as a slurm job. I think the 'permutation' part in CellChat takes time and perhaps needs parallelization.

To run Liana with my dataset and with more methods (requiring permutation), what would be your recommendation for the parameters in slurm?
Does this sound good or too much/less?
sbatch -t 11-00:00:00 -n 128 --mem=500g

Thank you.

Best,
Jiseok

[dbdimitrov]

Hi @redbybeing,

Honestly, I do not recall too well the rough requirements for liana in R. I would assume that 500gb is far more than enough for 100k cells.

Just for reference, 100k cells with 10 cell types in Python (liana+) take less than a couple of minutes (for most methods) and < 10gb (total peak memory, incl. the loaded object, session, etc).
https://www.nature.com/articles/s41556-024-01469-w/figures/6

liana+ should be much more efficient than liana(r) but I would assume that either way this should give you a clue.

Hope this helps.

Daniel

[redbybeing]

I see, thanks!
I'm sorry for asking another trivial question..

I tried running rscript in slurm, and I get a lot of errors like this:

Error: ℹ In index: 1.
ℹ With name: custom_resource.
Caused by error in `loadNamespace()`:
! there is no package called ‘Connectome’

When I ran the same script in Rstudio server, I got the Connectome rank result in the dataframe no problem.
Do you think it's because in slurm environment, somehow they cannot find the package?
I did specify library paths in my rscript like this though.

mylibpaths <- c(
  "/nas/longleaf/home/jiseokl/R/x86_64-pc-linux-gnu-library/4.3",
  "/nas/longleaf/home/jiseokl/R/x86_64-pc-linux-gnu-library/4.2",
  "/nas/longleaf/rhel8/apps/r/4.3.1/lib64/R/library"
)
.libPaths(mylibpaths)

Also at the final step, I get this error and the job halts without producing the final result dataframe.

7: Key ‘originalexp_’ taken, using ‘rna_’ instead 
Error in liana_res %>% map(function(res) nrow(res)) %>% as.numeric() : 
  'list' object cannot be coerced to type 'double'
Calls: %>% ... liana_aggregate -> %<>% -> %||% -> .select_cap -> %>%
In addition: Warning message:
In exec(output, ...) :
  Using internal and external methods should be done with caution!
Execution halted

Thank you for your time and input- I'm not used to using python so I'm trying the R Liana first..

[redbybeing]

Oh, actually I got to run everything on slurm up to liana_wrap() and saved the result before aggregating.
It was the liana_aggregate() step that's causing error.

> liana_res <- liana_res %>% liana_aggregate()
Error in liana_res %>% map(function(res) nrow(res)) %>% as.numeric() : 
  'list' object cannot be coerced to type 'double'
In addition: Warning message:
In exec(output, ...) :
  Using internal and external methods should be done with caution!

How could I fix this?

Thank you so much.