Description parsing behaviour for Markdown changed

[davidjgoss]

From v32, Gherkin is getting different results when parsing Markdown, specifically for the description.

This surfaced in the compatibility kit when trying to upgrade to v32:

• Update dependency @cucumber/gherkin to v32 cucumber-node#16
• https://github.com/cucumber/cucumber-node/actions/runs/13810695160/job/38631404644?pr=16

The critical parts are still fine - the same pickles come out compared to v31 - but there is now a description being picked up:

For reference the sample file being parsed is:
https://github.com/cucumber/compatibility-kit/blob/main/devkit/samples/markdown/markdown.feature.md

So it looks like we are picking up the first row of that table as a description, where before there was none.

[mpkorstanje]

Not entirely unexpected, the Markdown parser is most favorably described as a proof of concept bolted onto the Gherkin parser. And the problem seems to come from this:

The GherkinInMarkdownTokenMatcher will consider all lines that aren't recognised as a special token as Empty. The reason for this is that Markdown documents will typically have lines that have nothing to do with Gherkin - they are just prose.

Which is achieved by testing for the empty token:

gherkin/javascript/src/GherkinInMarkdownTokenMatcher.ts

Lines 97 to 110 in 2143371

	if (
	!this.match_TagLine(token) &&
	!this.match_FeatureLine(token) &&
	!this.match_ScenarioLine(token) &&
	!this.match_BackgroundLine(token) &&
	!this.match_ExamplesLine(token) &&
	!this.match_RuleLine(token) &&
	!this.match_TableRow(token) &&
	!this.match_Comment(token) &&
	!this.match_Language(token) &&
	!this.match_DocStringSeparator(token) &&
	!this.match_EOF(token) &&
	!this.match_StepLine(token)
	) {

And then that matches the comment line:

gherkin/javascript/src/GherkinInMarkdownTokenMatcher.ts

Lines 129 to 136 in 2143371

	match_Comment(token: Token): boolean {
	let result = false
	if (token.line.startsWith('|')) {
	const tableCells = token.line.getTableCells()
	if (this.isGfmTableSeparator(tableCells)) result = true
	}
	return this.setTokenMatched(token, null, result)
	}

Why, not a clue though.