WI #2232 Fix handling of pseudo keyword INCLUDE in EXEC SQL INCLUDE directives#2821
Merged
fm-117 merged 3 commits intoDec 9, 2025
Conversation
efr15
requested changes
Dec 8, 2025
efr15
approved these changes
Dec 8, 2025
fm-117
deleted the
2232-invalidoperationexception-in-selectalltokensbetween-method-2
branch
December 9, 2025 09:46
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2232
Ok this one needs an explanation...
The parser uses
TokenType.EXEC_SQLfor the word"INCLUDE"only when it appears after"EXEC SQL"."INCLUDE"is not a Cobol keyword, it should be considered as a user-defined word. However the parser needs to identify the beginning of all compiler directives so theEXEC_SQLtype is used for that purpose and is part of theCompilerDirectiveStartingKeywordtoken family."INCLUDE"works more or less like a contextual keyword.The exception we observed in
SelectAllTokensBetweencomes from the unfortunate choice of token type EXEC_SQL : when building dictionaries linking token text to token type and vice-versa, the "EXEC_SQL" type name is translated to "EXEC-SQL" text, meaning that when a user mispells "EXEC SQL", typing "EXEC-SQL" instead, the parser will create a token that is not there and then try to parse anEXEC SQL INCLUDEwhich does not exist. Furthermore, the parsing of EXEC SQL INCLUDE has a special mechanism to rewind two tokens back in the iterator because the "token" that starts the directive isINCLUDEbut the two previousEXECandSQLmust be included within the directive. Rewinding the iterator creates this inconsistent state where the stop token is located before the start token !Note that we have to explicitly delete the value for
EXEC_SQLin_TokenStringFromTokenTypebecause it has been added previously when translating all types to text, replacing'_'with'-'. The original code was actually doing nothing.The changes in
PreprocessorStep.csare rather an optimization than a fix, no need to try collecting tokens for a directive that could not be built. Still it can be useful to prevent other invalid selection interval problems.And finally I used an incremental test but the original exception should happen during full parsing too.