Replace markdown escaping with a combined md/Rd tokenizer

NEWS.md

@@ -1,5 +1,9 @@
 # roxygen2 (development version)
 
+* Markdown processing has been rewritten around a single tokenizer for the combined markdown/Rd grammar, replacing the old escape/unescape passes. Processing large documentation files is now substantially faster, and Rd tags behave consistently in every markdown context.
+* Markdown text containing `\%` now renders as `%`; previously it produced `\\%` in the Rd file, which truncated the displayed line at the `%`.
+* Markdown links with an escaped closing bracket (e.g. `[bar\]`) no longer leak a link reference definition into the generated Rd.
+* An Rd tag with an unterminated argument (e.g. `\code{a % b}`, where the unescaped `%` comments out the closing brace) now generates a warning instead of being silently reinterpreted as markdown.
 * Markdown processing now handles multibyte characters inside Rd tags correctly; previously a tag like `\code{café}` would corrupt the markdown interpretation of the text that followed it.
 * Markdown warnings triggered by a `rd_family_title` prefix (e.g. for an unsupported level 1 heading) no longer error.
 * Markdown link targets are now resolved against a per-run index of each package's help topics, instead of one `help()` call per topic and package. This substantially speeds up documenting packages with many cross-reference links, e.g. `roxygenize()` on testthat is about a third faster.

R/cpp11.R

@@ -24,6 +24,10 @@ find_includes <- function(path) {
   .Call(`_roxygen2_find_includes`, path)
 }
 
+tokenizeMd <- function(text, verbatim) {
+  .Call(`_roxygen2_tokenizeMd`, text, verbatim)
+}
+
 wrapUsage <- function(string, width, indent) {
   .Call(`_roxygen2_wrapUsage`, string, width, indent)
 }

R/markdown-link.R

@@ -70,8 +70,9 @@ get_md_linkrefs <- function(text) {
       paste0(
         "(?x)",
         "(?<=[^\\]\\\\]|^)", # must not be preceded by ] or \
-        "\\[([^\\]\\[]+)\\]", # match anything inside of []
-        "(?:\\[([^\\]\\[]+)\\])?", # match optional second pair of []
+        # match anything inside of [], not ending with \ (an escaped bracket)
+        "\\[([^\\]\\[]*[^\\]\\[\\\\])\\]",
+        "(?:\\[([^\\]\\[]*[^\\]\\[\\\\])\\])?", # optional second pair of []
         "(?=[^\\[{]|$)" # must not be followed by [ or {
       ),
       text,
@@ -101,6 +102,7 @@ parse_link <- function(destination, contents, state) {
     return(NULL)
   }
   destination <- sub("^R:", "", URLdecode(destination))
+  Encoding(destination) <- "UTF-8" # restore encoding dropped URLdecodse
 
   ## if contents is a `code tag`, then we need to move this outside
   is_code <- FALSE