How to call julia function inside of foreach and make it fast

[nremenyi]

I rewrote an R function in Julia to make it faster. I will call this function many times, and I thought about further parallelizing it in R and call the Julia function parallel.

When I do parallel run using “foreach/future”, the parallel loop fails, because the Julia function is not sourced to the threads / environments, I guess. If you add the sourcing to the loop, it gets really slow, obviously. Interestingly, after the foreach loop failed, if you run it again, the function works, but still kind of slow.

I am wondering if there is a proper way to source the function to the threads, and still make sure its fast, like i would execute it sequentially.

Some sample code:

Packages

library(JuliaCall)
library(microbenchmark)
library(future)
library(foreach)
library(doFuture)

Generate 210-by-3 matrix

set.seed(1)
m = matrix(sample.int(n = 210, size = 210, replace = FALSE), ncol = 3)

R function to shuffle (randomize) each row of a matrix

myFunc = function(m){
  out = apply(X = m, MARGIN = 1, sample)
  return(t(out))
}
myFunc(m)

Julia function to shuffle (randomize) each row of a matrix

writeLines("
using Random

function myFuncJL(; m::Matrix{Int64})
    for i in axes(m, 1)
       m[i,:] = Random.shuffle(m[i,:])
    end
    return m
end",
"myFuncJL.jl")
julia_source("myFuncJL.jl")
julia_call("myFuncJL", m = m)

function to load Julia and execute

myFuncJL_load = function(m){
  julia_source("myFuncJL.jl")
  m = julia_call("myFuncJL", m = m)
  return(m)
}
myFuncJL_load(m)

Benchmark (Julia is 10-20x faster)

mbm = microbenchmark::microbenchmark(
  r = myFunc(m),
  julia = julia_call("myFuncJL", m = m),
  times = 1000)
mbm

Call function 100 times in for loop (Julia faster)

n = 100
system.time({
  for(i in 1:n){
    myFunc(m)
  }
})
system.time({
  for(i in 1:n){
    julia_call("myFuncJL", m = m)
  }
})

Julia gets really slow with loading the function

system.time({
  for(i in 1:n){
    myFuncJL_load(m)
  }
})

Parallel execution of R on 4 cores

n_cores = 4
n = 1000
future::plan(future::multisession, workers = n_cores)
system.time({
  foreach::foreach (i = 1:n, .options.future = list(seed = TRUE)) %dofuture% {
    myFunc(m)
  }
})

This does not work, since function is not sources to parallel threads / environments
UndefVarError: myFuncJL not defined

system.time({
  foreach::foreach (i = 1:n, .options.future = list(seed = TRUE)) %dofuture% {
    julia_call("myFuncJL", m = m)
  }
})

Works but slow because we source function each time in loop

system.time({
  foreach::foreach (i = 1:n, .options.future = list(seed = TRUE)) %dofuture% {
    myFuncJL_load(m = m)
  }
})

Now works since function was loaded previously? Still kind of slow though

system.time({
  foreach::foreach (i = 1:n, .options.future = list(seed = TRUE)) %dofuture% {
    julia_call("myFuncJL", m = m)
  }
})

[HenrikBengtsson]

Hello. Before anything else, could you please re-format your comment to use a Code Block (see https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#quoting-code)