Skip to content

WI #2232 Fix handling of pseudo keyword INCLUDE in EXEC SQL INCLUDE directives#2821

Merged
fm-117 merged 3 commits into
developfrom
2232-invalidoperationexception-in-selectalltokensbetween-method-2
Dec 9, 2025
Merged

WI #2232 Fix handling of pseudo keyword INCLUDE in EXEC SQL INCLUDE directives#2821
fm-117 merged 3 commits into
developfrom
2232-invalidoperationexception-in-selectalltokensbetween-method-2

Conversation

@fm-117

@fm-117 fm-117 commented Dec 2, 2025

Copy link
Copy Markdown
Contributor

Fixes #2232

Ok this one needs an explanation...

The parser uses TokenType.EXEC_SQL for the word "INCLUDE" only when it appears after "EXEC SQL". "INCLUDE" is not a Cobol keyword, it should be considered as a user-defined word. However the parser needs to identify the beginning of all compiler directives so the EXEC_SQL type is used for that purpose and is part of the CompilerDirectiveStartingKeyword token family. "INCLUDE" works more or less like a contextual keyword.

The exception we observed in SelectAllTokensBetween comes from the unfortunate choice of token type EXEC_SQL : when building dictionaries linking token text to token type and vice-versa, the "EXEC_SQL" type name is translated to "EXEC-SQL" text, meaning that when a user mispells "EXEC SQL", typing "EXEC-SQL" instead, the parser will create a token that is not there and then try to parse an EXEC SQL INCLUDE which does not exist. Furthermore, the parsing of EXEC SQL INCLUDE has a special mechanism to rewind two tokens back in the iterator because the "token" that starts the directive is INCLUDE but the two previous EXEC and SQL must be included within the directive. Rewinding the iterator creates this inconsistent state where the stop token is located before the start token !

Note that we have to explicitly delete the value for EXEC_SQL in _TokenStringFromTokenType because it has been added previously when translating all types to text, replacing '_' with '-'. The original code was actually doing nothing.

The changes in PreprocessorStep.cs are rather an optimization than a fix, no need to try collecting tokens for a directive that could not be built. Still it can be useful to prevent other invalid selection interval problems.

And finally I used an incremental test but the original exception should happen during full parsing too.

Comment thread TypeCobol/Compiler/Preprocessor/PreprocessorStep.cs Outdated
Comment thread TypeCobol/Compiler/Preprocessor/PreprocessorStep.cs Outdated
@fm-117
fm-117 requested a review from efr15 December 8, 2025 13:30
@fm-117
fm-117 merged commit d81e26d into develop Dec 9, 2025
8 checks passed
@fm-117
fm-117 deleted the 2232-invalidoperationexception-in-selectalltokensbetween-method-2 branch December 9, 2025 09:46
@fm-117 fm-117 mentioned this pull request Dec 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

InvalidOperationException in SelectAllTokensBetween method

2 participants