[preprocessor] Fix crash/hang on truncated callable macro invocation

[EylonKrause]

Problem

The preprocessor crashes (SIGSEGV) or hangs on parseable SystemVerilog when a callable macro invocation is truncated at end-of-stream. Minimal reproducers (all with --expand_macros, e.g. via verible-verilog-preprocessor preprocess):

`define A(x) hello `A
`A(1)

`define A(x) x
`A            // top-level, no arguments

An included file ending in a callable macro reference (e.g. foo \A) crashes the same way, and a (with no matching)(``define A(x) hello `C( ``) spins in an infinite loop.

Root cause

GenerateBypassWhiteSpaces (verilog-preprocess.cc:63) pulls tokens from a StreamIteratorGenerator and dereferences the result — **iterator — with no end guard:

auto iterator = generator();
while (verilog::VerilogLexer::KeepSyntaxTreeTokens(**iterator) == 0) {
  iterator = generator();
}

MakeConstIteratorStreamer returns the view's end() iterator once exhausted (token-stream-adapter.h), so **iterator on end() is a double past-the-end dereference. The function relies on the invariant that every stream ends in a kept EOF token (KeepSyntaxTreeTokens(EOF) is true), which stops the loop and lets callers see isEOF(). The top-level analyzer stream satisfies this (it retains EOF), but the streams re-lexed for macro expansion do not: ExpandText, ExpandMacro, HandleInclude, and the standalone tool all build their token sequence with a loop that stops before the EOF (!lexer.GetLastToken().isEOF()). A truncated callable-macro invocation at the end of such a stream reaches ConsumeAndParseMacroCall, which calls GenerateBypassWhiteSpaces, whose generator() returns end() → crash. The sibling streamers (MakeTokenStreamer) already guard this by returning an EOFToken sentinel; GenerateBypassWhiteSpaces does not, because it only has the std::function, not the end iterator.

Fix

Restore the kept-EOF invariant on every re-lexed stream, so GenerateBypassWhiteSpaces stops at EOF and callers handle it gracefully (returning InvalidArgumentError) instead of dereferencing past the end:

• ExpandText / ExpandMacro / HandleInclude / the standalone tool: append the EOF token as an end sentinel before building the stream view.
• ExpandText / ExpandMacro token-pulling loops: break on the EOF sentinel so it is not forwarded into the expanded output.
• HandleInclude: skip the sentinel EOF when splicing the child stream into the parent (the child ScanStream forwards it as a pass-through token; without this it would land in the middle of the parent stream).
• ConsumeAndParseMacroCall argument loop: break on EOF. With the sentinel in place a ( without a matching ) would otherwise spin forever; this is the one caller that did not already handle a mid-scan EOF.
• ConsumeAndParseMacroCall: record the "illegal to call a callable macro without ()" error in preprocess_data_.errors (it was returned as a Status but never surfaced, so truncated input was silently accepted). This required making the method non-static (its only caller is non-static).

Testing

Built verible-verilog-preprocessor and confirmed all four crash variants and the open-paren hang now exit cleanly. The full //verible/verilog/preprocessor/... suite (including the existing "Nested callable macros" test) and verilog-analyzer_test pass. Added TruncatedCallableMacroDoesNotCrash covering the no-( (now a diagnostic) and open-( (now terminates) cases.

Out of scope (separate, pre-existing bug, not addressed here): unbounded macro self-recursion such as `define A(x) `A(x) overflows the stack with no depth guard — worth a follow-up.

---

Disclosure: this contribution was authored with an AI coding assistant (Claude) and reviewed before submission.

[hzeller]

The preprocessing test is failing the CI. Can you rebase (as there was some unrelated mac failure issue), and have a look at the failing test ?

[EylonKrause]

Rebased onto current master. The "preprocessing test failing" was the stale branch (it was ~4k commits behind) — after the rebase the whole verilog-preprocess_test suite, including this PR's TruncatedCallableMacroDoesNotCrash, passes locally, and it also picks up the unrelated macOS fix you mentioned.

One heads-up on overlap: this PR and #2532 both touch ConsumeAndParseMacroCall's end-of-stream handling, via different routes — #2535 makes the method non-static so it records the diagnostic itself, while #2532 keeps it static and has the caller record it. Whichever you'd rather land first, I'll rebase the other on top (or fold them together) so they don't collide — just say the word.