Allow to provide multiple filenames as `input` in `render_book()` to provide Rmd to build in the book project

[shirdekel]

I'd like to use both multiple file paths in the input argument to render_book() (in order to supply multiple dependencies for drake::knitr_in() in my drake plan), as well as specify output_format = "all". However, just doing the former gives me warnings that suggest that only the first file path is being used, and supplying both gives me an error. I struggled making a minimal reprex, so instead cloned the bookdown-demo repo and executed the below. I also found an rmarkdown issue that presents the same kind of error (rstudio/rmarkdown#1722), but I couldn't easily work out how it could apply here.

Works

bookdown::render_book('index.Rmd')

bookdown::render_book('index.Rmd', output_format = "all")

Produces warnings

bookdown::render_book(c('index.Rmd', '01-intro.Rmd'))

Warning messages:
1: In if ((n1 <- nchar(p)) == 0) return(x) :
  the condition has length > 1 and only the first element will be used
2: In if (is_sub_path(p, d, n2)) { :
  the condition has length > 1 and only the first element will be used
3: In if (p2 == "") p2 = "." :
  the condition has length > 1 and only the first element will be used

Produces error

bookdown::render_book(c('index.Rmd', '01-intro.Rmd'), output_format = "all")

Error in base::file(file, encoding = "UTF-8") : 
  invalid 'description' argument

Session info

R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14.6

Locale: en_AU.UTF-8 / en_AU.UTF-8 / en_AU.UTF-8 / C / en_AU.UTF-8 / en_AU.UTF-8

Package version:
  base64enc_0.1.3   bookdown_0.21.6   digest_0.6.27     evaluate_0.14    
  glue_1.4.2        graphics_4.0.2    grDevices_4.0.2   highr_0.8        
  htmltools_0.5.1.1 jsonlite_1.7.2    knitr_1.31        magrittr_2.0.1   
  markdown_1.1      methods_4.0.2     mime_0.10         rlang_0.4.10     
  rmarkdown_2.7.2   stats_4.0.2       stringi_1.5.3     stringr_1.4.0    
  tinytex_0.29      tools_4.0.2       utils_4.0.2       xfun_0.21        
  yaml_2.2.1       

---

By filing an issue to this repo, I promise that

• I have fully read the issue guide at https://yihui.org/issue/.
• I have provided the necessary information about my issue.
  • If I'm asking a question, I have already asked it on Stack Overflow or RStudio Community, waited for at least 24 hours, and included a link to my question there.
  • If I'm filing a bug report, I have included a minimal, self-contained, and reproducible example, and have also included xfun::session_info('bookdown'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version: remotes::install_github('rstudio/bookdown').
  • If I have posted the same issue elsewhere, I have also mentioned it in this issue.
• I have learned the Github Markdown syntax, and formatted my issue correctly.

I understand that my issue may be closed if I don't fulfill my promises.

[cderv]

Thanks for the report.

This error exists because currently you can't set output_format and use multiple inputs in render_book(). Using multiple inputs in render_book() is documented for preview = TRUE mainly. Usually you preview with only one format. About input:

For filenames, if preview = TRUE, only files specified in this argument are rendered, otherwise all R Markdown files specified by the book are rendered.

So currently, it is missing a check that would be if output_format is provided, input can be only a directory (in which case index.Rmd is used) or a single file. This would not allowed you to do what you are trying to do.

Here is some context in order to discuss your use case.

bookdown::render_book() works to render a book project, so a directory of files. render_book() needs to be provided with index.Rmd to have the information in the YAML and it will render all Rmd files in the directory, following configuration and rules. See https://bookdown.org/yihui/bookdown/usage.html
It will by default merged all the Rmd into one and render to HTML (M-K approach) or knit each Rmd and merge the result before rendering to HTML.

However, there is a special mode existing to support the Knit Button in the IDE: When you click the knit button, the file path is passed to render_site() which will call render_book(preview = TRUE). This will trigger a rerender only of those provided file in the book, all the other file won't be rendered. This works in both approachs mentioned above.

So passing multiple filenames in input is designed for preview mode of the book.
Is this what you are trying to do ? I am not sure... 🤔

I'd like to use both multiple file paths in the input argument to render_book() (in order to supply multiple dependencies for drake::knitr_in() in my drake plan), as well as specify output_format = "all"

We could also try to support or improve for new use case, but I am not sure to understand yours. Can you explain more what you want to do, and how you use drake and bookdown together ?
I think there is some thinking to do on how the two can be used together. I feel that, if I understand drake::knitr_in correctly, you will end up wrapping all your Rmd files in knitr_in() and pass them to render_book(input=) - is that the aim ?

I believe there are discussion already on how to best use drake and bookdown. see ropensci/drake#1337. There is also the new targets which also have the question ropensci/targets#261
Both approach try to couple the target approach of drake and targets by defining the dependencies to recompute before rendering the book. You may have to build around that so that you can use bookdown::render_book() as it is designed.

Thanks for bringing this topics.

(About the warnings, we got them because we made use of xfun::relative_path() which does not handle a vector of paths. That is an issue I'll fix, thanks).

[shirdekel]

Thanks for such a quick response! I'm glad the path vector issue was fixed.

My use case is compiling my PhD thesis. I have been using drake for experiment generation, data analyses, and visualisation, and now want to add a target that renders the final document using thesisdown. Ideally, I'd like this workflow to include two components: 1. Compilation to both PDF (for submission) and Word (for supervisor review), and 2. Dependency tracking of the individual .Rmd files.

I can achieve the first component fine by specifying output_format = "all" in bookdown::render_book() and the following in the index.Rmd YAML:

output:
  thesisdown::thesis_pdf: default 
  thesisdown::thesis_word: default

As for the second component, it was actually because of this example from @wlandau that I assumed that I'd need to specify multiple input arguments to bookdown::render_book() (through drake::knitr_in()) in order to make sure that all my .Rmd files are "tracked" by drake.

However, if this is not possible or not recommended, then Will's subsequent comment agrees with the OP's use of a deps argument in order to force dependency.

So I guess I can do something similar to this comment:

render_with_deps <- function(input, config_file, output_format, deps) {
  bookdown::render_book(
    input = input,
    config_file = config_file,
    output_format = output_format
  )
}

thesis_rmd <-
  c(
    "introduction.Rmd",
    "chapter_1.Rmd"
  )

the_plan <-
  drake_plan(
# ...
    thesis = render_with_deps(
      input = knitr_in("index.Rmd"),
      config_file = file_in("_bookdown.yml"),
      output_format = "all",
      deps = file_in(!!thesis_rmd)
    )
  )

[cderv]

I am not a drake power user myself, but it feel like this wrapper function render_with_deps() is a great way to achieve this. I would definitely recommend that !

For my knowledge, why use knitr_in() for index and file_in() for other rmd ?

FWIW, in last dev version, we improve a bit render_book() to take a directory, in which case index.Rmd will be look for, the book render in this directory.

This would mean that you could do (simplify ?):

render_with_deps <- function(input = ".", config_file, output_format, deps) {
  bookdown::render_book(
    input = input,
    config_file = config_file,
    output_format = output_format
  )
}
thesis_rmd <-
  c(
	"index.Rmd",
    "introduction.Rmd",
    "chapter_1.Rmd"
  )
the_plan <-
  drake_plan(
# ...
    thesis = render_with_deps(
      config_file = file_in("_bookdown.yml"),
      output_format = "all",
      deps = file_in(!!thesis_rmd)
    )
  )

To help with such tools, we could also maybe export the function that list the .Rmd files that will be used to build the bookdown project ? 🤔

You could also store it in rmd_files field in the config and retrieve it

thesis_rmd <- yaml::yaml_load.file("_bookdown.yml")[["rmd_files"]]

There is no exported function to get the config also.

Would that help or you would prefer anyway to have the list in your R file with your drake plan ?

[shirdekel]

First of all, I really appreciate the effort you're putting into this to help. Merci!

From what I understand, knitr_in() is like file_in() in that it declares the Rmd a dependency, but also declares a dependency for any targets that you mention inside the Rmd (using loadd() or readd()). So I'd need to use that for the list of Rmds.

I'd definitely prefer to only list the individual Rmds in the rmd_files of _bookdown.yml, rather then doing it again for the drake plan. So it could look like this:

# _bookdown.yml
rmd_files: 
  - "index.Rmd"
  - "introduction.Rmd",
  - "chapter_1.Rmd"

render_with_deps <- function(input = ".", config_file, output_format, deps) {
  bookdown::render_book(
    input = input,
    config_file = config_file,
    output_format = output_format
  )
}

Btw, seems that it's yaml::yaml.load_file() not yaml::yaml_load.file(). yaml::read_yaml() also works.

thesis_rmd <- yaml::yaml.load_file("_bookdown.yml")[["rmd_files"]]
the_plan <-
  drake_plan(
    # ...
    thesis = render_with_deps(
      config_file = file_in("_bookdown.yml"),
      output_format = "all",
      deps = knitr_in(!!thesis_rmd)
    )
  )

But can you clarify what exported function you are suggesting? Do you mean a function that is essentially a wrapper for yaml::yaml.load_file("_bookdown.yml")[["rmd_files"]]? I would definitely use it, but am also happy just to make my own wrapper using the yaml.load_file/read_yaml trick.

[cderv]

The render_with_deps() function seems the way to go with drake

But can you clarify what exported function you are suggesting?

Internally, we need to know what are the source Rmd file to build, and we need to load the configuration. Both of these functions could be useful in such case for external usage. Though, this require a bit of work to make one or the other exported, so i need to 🤔

However, listing files in _bookdown.yml, then use yaml::read_yaml() to get the information seems enough. It just won't deal with all possible customisation of bookdown (like using rmd_subdirs) - But for this usage with drake, maybe that is enough as you said.

I'll think about it. Thanks !

[shirdekel]

Thanks for the clarification. I could imagine that at some stage it might be preferable to list directories (e.g., "experiment_1/", which contains "experiment_1/introduction.Rmd", "experiment_1/method.Rmd", etc.) if there end up being a large enough number of individual Rmds. However, the above method seems to work fine for me now, thanks!

[cderv]

@yihui this is a rather long thread but I want to have your thoughts on the idea.

Tools like drake or targets needs to know the input file of a project to be able to build the dependency graph. Currently with a bookdown project there is no way to access for sure the files that will be rendered as part of the project. This happens internally in bookdown:::source_files().

One trick currently is to explicitly set the files in rmd_files field in YAML and read this, as bookdown will do. But this only works if rmd_subdirs is not used, and gets complicated if the rmd_files fields use different values per format.

I think it could be interesting to document and export a function like bookdown::source_files() so that one can retrieve the files that this bookdown project will read.

What do you think ? It seems not so difficult, and not harmful to export this internal function. This could help in this case.

Another additional solution would be to allow to pass a vector of filenames manually in render_book() - this would override the use of the internal source_files()

bookdown/R/render.R

Line 152 in f7d0622

files = source_files(format, config)

Either by allowing a vector of several files in input= that would be used as the files to render, or by adding an argument in render_book() to use in the above lines.