memory not mapped issue

[statquant]

It won't probably help you any onr bit but I am getting the following error when using the lattest fst trying to write different files in parallel (using foreach and doMC for the backhand)

INFO [2017-08-03 03:08:22] Writing [PATH_REDACTED]/perf_20170703.fst

 *** caught segfault ***
address 0x43ff000, cause 'memory not mapped'

Traceback:
 1: .Call("fst_fstretrieve", PACKAGE = "fst", fileName, columnSelection,     startRow, endRow)
 2: fstretrieve(fileName, columns, from, to)
 3: fstread(path, columns, from, to, as.data.table)
 4: read.fst(path = path, columns = columns, as.data.table = TRUE)
 5: readDataFile(path = inFilePath)
 6: convertFile(inFilePath, outFilePath, param)
 7: eval(expr, envir, enclos)
 8: eval(c.expr, envir = args, enclos = envir)
 9: doTryCatch(return(expr), name, parentenv, handler)
10: tryCatchOne(expr, names, parentenv, handlers[[1L]])
11: tryCatchList(expr, classes, parentenv, handlers)
12: tryCatch(eval(c.expr, envir = args, enclos = envir), error = function(e) e)
13: FUN(X[[i]], ...)
14: lapply(X = S, FUN = FUN, ...)
15: doTryCatch(return(expr), name, parentenv, handler)
16: tryCatchOne(expr, names, parentenv, handlers[[1L]])
17: tryCatchList(expr, classes, parentenv, handlers)
18: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call)[1L]        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        msg <- conditionMessage(e)        sm <- strsplit(msg, "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && identical(getOption("show.error.messages"),         TRUE)) {        cat(msg, file = stderr())        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})
19: try(lapply(X = S, FUN = FUN, ...), silent = TRUE)
20: sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE))
21: FUN(X[[i]], ...)
22: lapply(seq_len(cores), inner.do)
23: mclapply(argsList, FUN, mc.preschedule = preschedule, mc.set.seed = set.seed,     mc.silent = silent, mc.cores = cores)
24: e$fun(obj, substitute(ex), parent.frame(), e$data)
25: foreach(inFilePath = inFileList, outFilePath = outFileList) %dopar%     convertFile(inFilePath, outFilePath, param)
26: convertFiles(param)
27: run(param = opt)
An irrecoverable exception occurred. R is aborting now ...

[MarcusKlik]

Hi @statquant, thanks for reporting the issue. I'm taking special measures to disable OpenMP multithreading when fst is used from multiple parallel R instances and I suspect something might go wrong there. It might also be an R memory problem like suggested here.

It might take some testing to get to the bottom of this one, thanks again for taking the time to report!

[renkun-ken]

Using OpenMP in forked R sessions created by mcfork may end up in deadlock. See Rdatatable/data.table#2031.

[MarcusKlik]

Hi @renkun-ken, thanks for pointing out that very relevant issue! Indeed, fst will run the same risk on a deadlock when called from a fork. So I have basically followed the same recipe as in the data.table package with a call to pthread_atfork from the package initialization method. This method is then used to set the number of threads to 1 when called from a fork and later to restore the previous number of threads when the fork ends. That code can be found here.

But given the error that @statquant's encountered during parallel operations, I guess more testing is needed to see if the implementation is solid the way it is now!

[MarcusKlik]

Hi @renkun-ken, a small test on an Ubuntu 16 .04 machine reveals:

library(parallel)

attach_fst <- function(nodeID)
{
  require(fst)
  
  fstgetthreads()
}

# number of threads on master
attach_fst(0)
#> Loading required package: fst
#> [1] 8

# number of threads on a fork cluster
cluster <- makeForkCluster(4)
parSapply(cluster, 1:4, attach_fst)
#> [1] 1 1 1 1
stopCluster(cluster)

# number of threads on a PSOCK cluster
cluster <- makePSOCKcluster(4)
parSapply(cluster, 1:4, attach_fst)
#> [1] 8 8 8 8
stopCluster(cluster)

So the fork cluster is scaled down in threads to a single thread as expected by executing the pthread_atfork function. The PSOCK cluster is not automatically scaled down however. And although that is intended behavior (also in the data.table package if I'm not mistaken), now that I'm thinking about it, that might lead to an unsuspecting user using a lot of threads (in this case 8 * 8 if all threads were used on a 8 node cluster). It would be better to have a method of detecting the package running on a PSOCK node as well.

So obviously there is some more thinking to be done :-), thanks for getting me in that direction!
(and great work on your packages by the way)

[MarcusKlik]

Hi @statquant, just checking in to ask you if you still get this error when using fst on a parallel cluster?

In the latest dev version, I've added better error handling for the C++ back-end library, so I'm curious to see if we can get a better idea of the problems you're seeing. Would it be possible to post a reproducible example that generates this error? thanks!