Replies: 1 comment 4 replies
-
|
Hello. TL;DRThe following uses best practices in R to capture output sent to standard output and messages to one file per iteration: Personally, I would tidy it up by using a helper function that I can reuse, e.g. Note that this is the approach you should take regardless of parallelization or not, e.g. with y <- list()
for(i in 1:2) {
redirect_stdout_and_messages_to_file(paste0('testout', i, '.log'), {
nimble::messageIfVerbose('nimble message') # message from Nimble
message('message message')
cat('cat message\n')
})
y[[i]] <- i
}DetailsThere are a few things going on in your original example. I'll try to clarify what's going, that things are working as expected and by design, and how you can get what you want. For a started, yes, correct res <- tryCatch(nimble::messageIfVerbose('nimble message'), condition = identity)
str(res)
#> List of 2
#> $ message: chr "nimble message\n"
#> $ call : language message(...)
#> - attr(*, "class")= chr [1:3] "simpleMessage" "message" "condition"So, in that sense, and for the discussion here, Second, you can think of Third, output from Fourth, the correct way to capture messages is not analogue to howe we capture standard output. That is, we do not want to use |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the explanations. I think I simplified my use case too much: the problem, which I'm not managing to achieve with The During the computation, output (diagnostic) from With library('doParallel')
library('foreach')
cl <- parallel::makeCluster(2)
doParallel::registerDoParallel(cl)
foreach(i = 1:2) %dopar% {
outcon <- file(paste0('testout', i, '.log'), open = 'w')
sink(outcon, type = 'output')
sink(outcon, type = 'message')
t = 0
while(t < 2) { # some convergence condition
## some Monte Carlo computations
message('log message\n') # these come from package Nimble
cat('log cat\n')
## some computations, need to inform main R session
sink(NULL, type = 'message')
message('message to main R session\n')
sink(outcon, type = 'message')
## more computations
message('another log message\n') # from package Nimble
cat('another log cat\n')
t = t + 1
}
sink(NULL, type = 'output')
sink(NULL, type = 'message')
close(outcon)
## some output here
}
foreach::registerDoSEQ()
parallel::stopCluster(cl)But I don't manage to achieve the same when I replace I've also tried This is the (proto)package in question. The At the moment the |
Beta Was this translation helpful? Give feedback.
-
|
I see. I'd like to clarify that when doing: library(foreach)
doParallel::registerDoParallel(2)
y <- foreach(x=1:2) %dopar% {
message("Hello")
print(x)
x
}it's a hit-and-miss whether you, or the user, will actually see any of the output produced. If I run the above using $ R --vanilla --quiet
> source("above-script.R")
Hello
[1] 1
Hello
[1] 2
>But if I run the same in the Windows terminal, RStudio Console, or the Windows RGui, there is no output displayed what so ever: > source("above-script.R")
>That you see output is more or less another one of those "lucky" coincidents. You have the same behavior with library(foreach)
doParallel::registerDoParallel(2)
output <- utils::capture.output({
y <- foreach(x=1:2) %dopar% {
print(x)
x
}
})you'll probable see nothing at all, and > output
character(0)Point is, you cannot reliably use your current approach and expect it to work everywhere. This is one of many things that I'm trying to address in Futureverse. The following will work however and wherever you run R, and regardless of parallel backend: library(foreach)
library(doFuture)
plan(multisession, workers = 2)
output <- utils::capture.output({
y <- foreach(x=1:2) %dofuture% {
print(x)
x
}
})
print(output)
#> [1] "[1] 1" "[1] 2"Now, with the above in mind, if we revisit your current approach: ## some computations, need to inform main R session
sink(NULL, type = 'message')
message('message to main R session\n')
sink(outcon, type = 'message')It's important to understand that that Now, because for you want to direct some of the messages to file and others not, and we cannot distinguish them by class, that is, they are all of class Are you in control of those message that you want to pass through? If so, I would introduce: immediateMessage <- function(..., domain = NULL, appendLF = TRUE) {
msg <- .makeMessage(..., domain = domain, appendLF = appendLF)
call <- sys.call()
m <- simpleMessage(msg, call)
class(m) <- c(class(m), "immediateCondition")
message(m)
invisible(m)
}and the update: redirect_stdout_and_messages_to_file <- function(file, expr, envir = parent.frame()) {
expr <- substitute(expr)
## Send all standard output to file
outcon <- file(file, open = "wt")
sink(outcon, type = "output")
on.exit({
sink(NULL, type = "output")
close(outcon)
})
withCallingHandlers({
eval(expr, envir = envir)
}, message = function(msg) {
if (!inherits(msg, "immediateCondition")) {
## Output message to stdout, so they end up in the file
cat(conditionMessage(msg))
## Prevent messages from being relayed to the main R session
invokeRestart("muffleMessage")
}
})
} ## redirect_stdout_and_messages_to_file()Then, whenever you call: immediateMessage("message to main R session")it will be relayed by the Futureverse back to the main R session, and in a near-live fashion. So, with the above two help functions, you'd can do: ## Configure parallelization
plan(multisession, workers = 2)
y <- foreach(i = 1:2) %dofuture% {
redirect_stdout_and_messages_to_file(paste0('testout', i, '.log'), {
t <- 0
while(t < 2) { # some convergence condition
## some Monte Carlo computations
message('log message') # these come from package Nimble
cat('log cat\n')
## some computations, need to inform main R session
immediateMessage('message to main R session')
## more computations
message('another log message\n') # from package Nimble
cat('another log cat\n')
t <- t + 1
}
})
}Again, this might look convoluted, but again, this is really how you do "capture everything, but ...", and this regardlessly of parallelization or not. Here are some references in my useR 2022 tutorial 'An Introduction to Futureverse for Parallel Processing in R', if you want to dive into the details:
PS. You don't need to add |
Beta Was this translation helpful? Give feedback.
-
|
This is extremely helpful, thank you so much! 🙏 Let me try to implement this, I can get back to you about the result if you're interested. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, let me, and current and future followers, know how it goes. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I'm calling the package
Nimblefrom aforeach()loop with%dofuture%. Nimble outputs some messages, which I'd like to be redirected to log files, together with other messages generated withcat(). Nimble seems to be using themessage()function, here is its codeI try to achieve this by opening a connection to the log file and redirecting
sink()there. This approach worked with thedoParallelbackend, but thefuturebackend keeps on sending the messages from Nimble ormessage()to the main R session.Is this a bug? Or should I approach my goal differently?
Example with
future:in this case the "cat message" correctly appears in each
testout....logfile, but the "nimble message" and the output frommessage()are displayed in the main R session.Compare with
doParallel:In this case all three messages appear in the log file.
I would expect the code with
futureto behave in a similar way as the one withdoParallel, redirecting both messages to the log file.Session information
Beta Was this translation helpful? Give feedback.
All reactions