strange behavior of the `markdown_chunk()` function

[agricolamz]

Hello,

I found strange behavior of the markdown_chunk() function. It looks like it stop dividing markdown into sections if it finds some strange symbols:

library(tidyverse)
library(ragnar)

read_lines("works_fine.md") |> 
  markdown_chunk(md = _, target_size = NA, segment_by_heading_levels = 1:6) |> 
  as_tibble() 

# A tibble: 3 × 4                                                                                              
  start   end context text                                                  
  <int> <int> <chr>   <chr>                                                 
1     1   672 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, magna urna …
2   673  1348 ""      "## Look at me\n\nLorem ipsum dolor sit amet, magna u…
3  1349  2019 ""      "## TEST 2\n\nLorem ipsum dolor sit amet, magna urna …

read_lines("works_strange.md") |> 
  markdown_chunk(md = _, target_size = NA, segment_by_heading_levels = 1:6) |> 
  as_tibble() 

# A tibble: 2 × 4                                                                                              
  start   end context text                                                  
  <int> <int> <chr>   <chr>                                                 
1     1   672 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, magna urna …
2   673  1454 ""      "## Look at me\n\nLong [aː], [ãː], [eː], [ẽː] are esp…

• works_fine.md:

## TEST 1

Lorem ipsum dolor sit amet, magna urna ut malesuada vitae feugiat et odio enim morbi arcu. Ut vel, sed mi, venenatis a. Donec ornare vulputate est, maecenas natoque orci. Molestie sapien vel non arcu justo vel dui inceptos. Etiam porta sit sed taciti lacus, tempus fusce. Vivamus eros at sed fusce penatibus sed ultrices justo. Sapien porttitor condimentum platea nec amet vehicula accumsan. Vel et nunc eget libero convallis. Sit volutpat vestibulum et in nascetur laoreet, vestibulum, risus. Non eget augue iaculis netus taciti non nunc, in conubia. Amet rutrum sed nec consectetur. Efficitur id, et rhoncus, et nam. Amet aliquam euismod et sagittis ligula.

## Look at me

Lorem ipsum dolor sit amet, magna urna ut malesuada vitae feugiat et odio enim morbi arcu. Ut vel, sed mi, venenatis a. Donec ornare vulputate est, maecenas natoque orci. Molestie sapien vel non arcu justo vel dui inceptos. Etiam porta sit sed taciti lacus, tempus fusce. Vivamus eros at sed fusce penatibus sed ultrices justo. Sapien porttitor condimentum platea nec amet vehicula accumsan. Vel et nunc eget libero convallis. Sit volutpat vestibulum et in nascetur laoreet, vestibulum, risus. Non eget augue iaculis netus taciti non nunc, in conubia. Amet rutrum sed nec consectetur. Efficitur id, et rhoncus, et nam. Amet aliquam euismod et sagittis ligula.

## TEST 2

Lorem ipsum dolor sit amet, magna urna ut malesuada vitae feugiat et odio enim morbi arcu. Ut vel, sed mi, venenatis a. Donec ornare vulputate est, maecenas natoque orci. Molestie sapien vel non arcu justo vel dui inceptos. Etiam porta sit sed taciti lacus, tempus fusce. Vivamus eros at sed fusce penatibus sed ultrices justo. Sapien porttitor condimentum platea nec amet vehicula accumsan. Vel et nunc eget libero convallis. Sit volutpat vestibulum et in nascetur laoreet, vestibulum, risus. Non eget augue iaculis netus taciti non nunc, in conubia. Amet rutrum sed nec consectetur. Efficitur id, et rhoncus, et nam. Amet aliquam euismod et sagittis ligula.

• works_strange.md:

## TEST 1

Lorem ipsum dolor sit amet, magna urna ut malesuada vitae feugiat et odio enim morbi arcu. Ut vel, sed mi, venenatis a. Donec ornare vulputate est, maecenas natoque orci. Molestie sapien vel non arcu justo vel dui inceptos. Etiam porta sit sed taciti lacus, tempus fusce. Vivamus eros at sed fusce penatibus sed ultrices justo. Sapien porttitor condimentum platea nec amet vehicula accumsan. Vel et nunc eget libero convallis. Sit volutpat vestibulum et in nascetur laoreet, vestibulum, risus. Non eget augue iaculis netus taciti non nunc, in conubia. Amet rutrum sed nec consectetur. Efficitur id, et rhoncus, et nam. Amet aliquam euismod et sagittis ligula.

## Look at me

Long [aː], [ãː], [eː], [ẽː] are especially frequent in verb forms, in particular causatives.

## TEST 2

Lorem ipsum dolor sit amet, magna urna ut malesuada vitae feugiat et odio enim morbi arcu. Ut vel, sed mi, venenatis a. Donec ornare vulputate est, maecenas natoque orci. Molestie sapien vel non arcu justo vel dui inceptos. Etiam porta sit sed taciti lacus, tempus fusce. Vivamus eros at sed fusce penatibus sed ultrices justo. Sapien porttitor condimentum platea nec amet vehicula accumsan. Vel et nunc eget libero convallis. Sit volutpat vestibulum et in nascetur laoreet, vestibulum, risus. Non eget augue iaculis netus taciti non nunc, in conubia. Amet rutrum sed nec consectetur. Efficitur id, et rhoncus, et nam. Amet aliquam euismod et sagittis ligula.

Cheers!

[t-kalinowski]

Hi,

Thank you for the issue! If I modify your MRE slightly, to test just the presence of non-ASCII characters, I don't see any issues. It looks like we chunk exactly the same way. Are you perhaps on Windows, and the locale is different? That's my best hypothesis for why you might be seeing this behavior.

Here is the updated MRE.

library(readr)
library(tibble)
library(ragnar)

md_base <- paste(
  "## TEST 1",
  stringi::stri_rand_lipsum(1),
  "## Look at me",
  stringi::stri_rand_lipsum(1),
  "## TEST 2",
  stringi::stri_rand_lipsum(1),
  sep = "\n\n"
)

# Find the start of the paragraph immediately after "## Look at me"
m <- regexpr(
  "(?<=## Look at me\\n\\n).+",
  md_base,
  perl = TRUE
)

md_strange <- md_base

replacement <- "Long [aː], [ãː], [eː], [ẽː] are especially frequent in verb forms."
start <- as.integer(m)
end <- start + nchar(replacement) - 1

substring(md_strange, start, end) <- replacement

f1 <- tempfile(fileext = ".md")
f2 <- tempfile(fileext = ".md")
writeLines(md_base, f1, useBytes = TRUE)
writeLines(md_strange, f2, useBytes = TRUE)


read_lines(f1) |>
  markdown_chunk(target_size = NA, segment_by_heading_levels = 1:6) |>
  as_tibble()
#> # A tibble: 3 × 4
#>   start   end context text                                                      
#>   <int> <int> <chr>   <chr>                                                     
#> 1     1   706 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, dui ut pellente…
#> 2   707  1040 ""      "## Look at me\n\nLorem ipsum dolor sit amet, et ridiculu…
#> 3  1041  1558 ""      "## TEST 2\n\nLorem ipsum dolor sit amet, tempus tempor n…

read_lines(f2) |>
  markdown_chunk(target_size = NA, segment_by_heading_levels = 1:6) |>
  as_tibble()
#> # A tibble: 3 × 4
#>   start   end context text                                                      
#>   <int> <int> <chr>   <chr>                                                     
#> 1     1   706 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, dui ut pellente…
#> 2   707  1040 ""      "## Look at me\n\nLong [aː], [ãː], [eː], [ẽː] are especia…
#> 3  1041  1558 ""      "## TEST 2\n\nLorem ipsum dolor sit amet, tempus tempor n…

^{Created on 2026-01-16 with reprex v2.1.1}

[agricolamz]

Hi, @t-kalinowski,

thank you for your answer! For some reason it left me puzzled... First, I'd like to assure you that I'm on Linux. I still think there is a problem...

library(readr)
library(tibble)
library(ragnar)

your_string <- "Long [aː], [ãː], [eː], [ẽː] are especially frequent in verb forms."
magic_line <- "Long [aː], [ãː], [eː], [ẽː] are especially frequent in verb forms."

md_base <- paste(
  "## TEST 1",
  stringi::stri_rand_lipsum(1),
  "## Look at me",
  stringi::stri_rand_lipsum(1),
  "## TEST 2",
  stringi::stri_rand_lipsum(1),
  sep = "\n\n"
)

md_strange1 <- paste(
  "## TEST 1",
  stringi::stri_rand_lipsum(1),
  "## Look at me",
  your_string,
  "## TEST 2",
  stringi::stri_rand_lipsum(1),
  sep = "\n\n"
)

md_strange2 <- paste(
  "## TEST 1",
  stringi::stri_rand_lipsum(1),
  "## Look at me",
  magic_line,
  "## TEST 2",
  stringi::stri_rand_lipsum(1),
  sep = "\n\n"
)

f1 <- tempfile(fileext = ".md")
f2 <- tempfile(fileext = ".md")
f3 <- tempfile(fileext = ".md")

writeLines(md_base, f1, useBytes = TRUE)
writeLines(md_strange1, f2, useBytes = TRUE)
writeLines(md_strange2, f3, useBytes = TRUE)

read_lines(f1) |>
  markdown_chunk(target_size = NA, segment_by_heading_levels = 1:6) |>
  as_tibble()
#> # A tibble: 3 × 4
#>   start   end context text                                                      
#>   <int> <int> <chr>   <chr>                                                     
#> 1     1   546 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, montes libero s…
#> 2   547  1154 ""      "## Look at me\n\nLorem ipsum dolor sit amet, at habitass…
#> 3  1155  1820 ""      "## TEST 2\n\nLorem ipsum dolor sit amet, tellus dignissi…

read_lines(f2) |>
  markdown_chunk(target_size = NA, segment_by_heading_levels = 1:6) |>
  as_tibble()
#> # A tibble: 3 × 4
#>   start   end context text                                                      
#>   <int> <int> <chr>   <chr>                                                     
#> 1     1   454 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, nec aenean duis…
#> 2   455   537 ""      "## Look at me\n\nLong [aː], [ãː], [eː], [ẽː] are especia…
#> 3   538  1012 ""      "## TEST 2\n\nLorem ipsum dolor sit amet, platea class ri…

read_lines(f3) |>
  markdown_chunk(target_size = NA, segment_by_heading_levels = 1:6) |>
  as_tibble()
#> # A tibble: 2 × 4
#>   start   end context text                                                      
#>   <int> <int> <chr>   <chr>                                                     
#> 1     1   580 ""      "## TEST 1\n\nLorem ipsum dolor sit amet, varius tellus j…
#> 2   581  1337 ""      "## Look at me\n\nLong [aː], [ãː], [eː], [ẽː] are especia…

^{Created on 2026-01-18 with reprex v2.1.1}

I've made some digging, however I don't know what to do with it. Here is my clumsy code:

library(tidyverse)

  your_string <- "Long [aː], [ãː], [eː], [ẽː] are especially frequent in verb forms."
  magic_line <- "Long [aː], [ãː], [eː], [ẽː] are especially frequent in verb forms."
  
cbind(your_string_split = stringr::str_split(your_string, "")[[1]],
      magic_line_split = stringr::str_split(magic_line, "")[[1]]) |> 
  as_tibble() |> 
  filter(your_string_split != magic_line_split) ->
  result

result
#> # A tibble: 2 × 2
#>   your_string_split magic_line_split
#>   <chr>             <chr>           
#> 1 ã                 ã               
#> 2 ẽ                 ẽ

charToRaw(result$your_string_split[1])
#> [1] c3 a3
charToRaw(result$magic_line_split[1])
#> [1] 61 cc 83

charToRaw(result$your_string_split[2])
#> [1] e1 ba bd
charToRaw(result$magic_line_split[2])
#> [1] 65 cc 83

^{Created on 2026-01-18 with reprex v2.1.1}

Looking at the result of charToRaw I can stipulate that your_string line is using some merged symbols and magic_line uses combining diacritics, but I don't know, how to prove it. This is the first time I am trying to analyze raw bytes...