Conversation
|
This fix is probably not ok; I originally had a Token::SetKind method in the Token class, and the reviewer wanted me to remove it, in favor of just creating new tokens when needed instead. I'm not sure I fully understand the problem that you are trying to solve here; could you elaborate a bit more please? |
|
More info (after discussing this in the meeting -- just noting down what we said): If splitting the token occurs inside a tentative parsing block, we need to do something like make a tentative copy of the lexer's m_lexed_tokens vector, and do the splitting/inserting there. If we decide to keep the tentative parsing, then we would replace the m_lexed_tokens vector in the lexer with our tentative copy. If we don't do something like this, then we could end up with m_lexed_tokens in the lexer containing an extra token (for each tentative parsing attempt that got rolled back), which could lead to some very confusing errors. |
Change current token kind in the token list instead of a current copy, because if a
TentativeParsingRollbackhappens, it discards the changed copy.