Terminating build part way through book leaves many intermediate files lying around

[hadley]

e.g. all of the .md files of the chapters built so far, and a number of render*.rds

[cderv]

By Terminating build part part way through book, you are meaning stopping the build ? Or after an error occurs ?

I'll need to try reproduce this in order to see what is happening.

BTW there is a bookdown::clean_book() function available - but this may not be for the files you are seing

[hadley]

Generally, I mean pressing cmd + c. I suspect the problem is that the cleanup code is just run after the book is successfully built, rather than being wrapped in an on.exit().

[cderv]

There are a bunch of on.exit() call already. For example, the one for the rds file:

bookdown/R/render.R

Lines 168 to 169 in 7ace59f

	render_args = tempfile('render', '.', '.rds')
	on.exit(file.remove(render_args), add = TRUE)

I'll try to reproduce and see why the cleaning is not happening

[yihui]

In theory, these files should be move/cleaned since I did use on.exit() to do that. Unless R crashed very badly, it shouldn't leave the intermediate files in the root dir.

[hadley]

Hmmm, weird. I double checked that you always used add = TRUE (since that's another easy mistake) but it looked ok to me.

Another thing worth checking if it you're using relative paths and the working directory changes at some point.

[cderv]

Yes in theory they should be removed. However, while investigating #920 , I stumble upon this issue with intermediary md files still present. If there is an error in the book (using new_session: true), the rendering will stop but there still be the .md file present in the directory. They are not deleted.

Reprex:

dir.create(temp_proj <- tempfile())
old <- setwd(temp_proj)
bookdown:::bookdown_skeleton('.')
#> [1] TRUE
files1 <- fs::dir_ls()
files1
#> 01-intro.Rmd       02-literature.Rmd  03-method.Rmd      04-application.Rmd 
#> 05-summary.Rmd     06-references.Rmd  book.bib           index.Rmd          
#> preamble.tex       README.md          style.css          _bookdown.yml      
#> _output.yml
content <- xfun::read_utf8("02-literature.Rmd")
content <- c(content, "```{r}", "stop('Will be an error')", '```')
xfun::write_utf8(content, "02-literature.Rmd")
yml <- yaml::read_yaml("_bookdown.yml")
yml <- c(yml, list(new_session = TRUE))
yaml::write_yaml(yml, "_bookdown.yml")
bookdown::render_book("index.Rmd", "bookdown::gitbook")
#> Error in Rscript_render(f, render_args, render_meta, add1, add2): Failed to compile 02-literature.Rmd
files2 <- fs::dir_ls()
setdiff(files2, files1)
#> [1] "01-intro.md"           "01-intro.utf8.md"      "02-literature.knit.md"
#> [4] "index.md"              "index.utf8.md"         "packages.bib"         
#> [7] "_bookdown_files"
setwd(old)
unlink(temp_proj)

However, the .rds file is not present so I guess there is some place where this does not go well and other (like which rds) which are ok.

I'll double check this tomorrow, specificaly the relative path.

[jtbayly]

I often have extra files left when the build fails because of an error, fwiw. (I make a lot of errors. lol)

[hadley]

The problem is that the intermediate files are left behind by:

for (f in files[rerun]) Rscript_render(f, render_args, render_meta, add1, add2)

And those keep intermediate files because clean is explicitly set to FALSE:

saveRDS(
    list(output_format = output_format, ..., clean = FALSE, envir = envir),
    render_args
  )

[yihui]

1. The intermediate file render*.rds is cleaned up at

   bookdown/R/render.R

   Line 191 in ca0145f

   on.exit(file.remove(render_args), add = TRUE)

   If R is terminated by clicking the red Stop button in RStudio, for some reason this on.exit() is not called. @hadley I've verified that even with your fix in Improved intermediate cleanup #1079, the .rds will still be left inside the tempdir(). We need to figure out why the Stop button in RStudio kills R so brutally.
2. For the intermediate *.md files, they are left there if a certain Rmd fails to compile, but they will be cleaned up again the next time after you fix the error in the failing Rmd document and recompile it:

   bookdown/R/render.R

   Lines 222 to 223 in ca0145f

   	intermediates = unlist(lapply(meta, attr, 'intermediates', exact = TRUE))
   	if (clean) on.exit(unlink(intermediates, recursive = TRUE), add = TRUE)

[juliasilge]

I find that I am left with many intermediate files (.md files, render.rds files, etc) when a bookdown render fails due to an error and that I have to manually delete them, or else the next time I try to render() it fails again. I have assumed this is because of being in an uncomfortable cached situation.

I am not clicking the red 🛑 RStudio button; the render is erroring out part of the way through.

[yihui]

I find that I am left with many intermediate files (.md files, render.rds files, etc) when a bookdown render fails due to an error and that I have to manually delete them, or else the next time I try to render() it fails again.

Hmm... After you fix the error in the Rmd, bookdown should clean up the intermediate files the next time you build the book. I wasn't able to reproduce this problem the last time, but let me try again.

[yihui]

Unfortunately I still can't reproduce the problem. Here I add an error chunk to index.Rmd:

d = tempfile()
dir.create(d)
owd = setwd(d)
bookdown:::bookdown_skeleton('test')
setwd('test')

cat('new_session: true\n', file = '_bookdown.yml', append = TRUE)
cat('```{r}
stop("Error")
```
', file = 'index.Rmd', append = TRUE)

bookdown::render_book()

Rendering index.Rmd will fail, and leave an intermediate index.knit.md:

cat(list.files(), sep = '\n')

_bookdown.yml
_output.yml
01-intro.Rmd
02-literature.Rmd
03-method.Rmd
04-application.Rmd
05-summary.Rmd
06-references.Rmd
book.bib
index.knit.md  # <- the intermediate file
index.Rmd
packages.bib
preamble.tex
README.md
style.css

Then I remove the error chunk and re-render:

xfun::process_file('index.Rmd', function(x) head(x, -3))
bookdown::render_book()
cat(list.files(), sep = '\n')

_book
_bookdown_files
_bookdown.yml
_output.yml
01-intro.Rmd
02-literature.Rmd
03-method.Rmd
04-application.Rmd
05-summary.Rmd
06-references.Rmd
book.bib
index.Rmd
packages.bib
preamble.tex
README.md
style.css
test.rds

The intermediate .knit.md file is gone.

Same thing when building the book with the RStudio IDE.

So I still need a reproducible example...

[juliasilge]

Let me see if I can put one together soon; I think the problem may be related to how caching is working when some chunks depend on something that has errored.