include Lua filter [DRAFT]

[kainctl]

Warning

This is still a WIP and I might be missing something important but the initial tests seem quite promising!

Thanks to the amazing work of @tarleb in #11371 it should now (in the latest master branch) be possible to create a full include Lua filter.

The following Lua filter can inject the contents of one file into the current document, while correctly resolving the media files/without breaking --extract-media:

include.lua

local pandoc = require 'pandoc'

local function include(code_block)
  if code_block.classes[1] ~= "include" then
    return nil
  end

  local format = "markdown"
  if code_block.classes[2] then
    format = code_block.classes[2]
  end

  local result_blocks = pandoc.Blocks{}

  for line in code_block.text:gmatch("[^\r\n]+") do
    local file_path = line:match("%S+")
    if file_path then
      local data = pandoc.system.read_file(file_path)
      local dir_path = pandoc.path.directory(file_path)
      -- I need to change the working directory, as otherwise
      -- nested includes across directories would _not_ work.
      -- Though in these cases, the resource path should always be
      -- the current working directory.
      local blocks = pandoc.system.with_working_directory(dir_path, function()
        return pandoc.with_state({resource_path = {'.'}}, function()
          local doc = pandoc.read(data, format)
          -- TODO: Mediabag might require some patching to avoid
          -- "clashes" with subdirectories of nested include calls
          -- that have the same file names, since the directory paths
          -- would be "dropped".
          pandoc.mediabag.fill(doc)
          local res = Pandoc(doc)
          return res.blocks
        end)
      end)
      result_blocks:extend(blocks)
    end
  end

  return result_blocks
end

function Pandoc(doc)
  return doc:walk{
    CodeBlock = include
  }
end

Calling it would look like:

Injecting content from a different file

```include
./subdir/a.md
```

Any pandoc supported format can be used (defaults to `markdown`)

```include {.org}
./subdir/b.org
```

[tarleb]

There is the include-files filter; any PR there that improves the current behavior, adds new features, or supports additional syntax would be most welcome!

[kainctl]

Nice! I wasn't aware of this filter. I did not consider adjusting the headings, that makes a lot of sense.

[kainctl]

Thank you again for the pointer! Reading through that source code was very insightful.
There is nothing that I can add; it is objectively the better way to do it.

I will use that instead and if I find anything that could be improved, I will submit a PR. 🙏