e67f4d5a92
The tokenizer already broke multi-word identifiers when the NEXT word was a keyword, but not when the identifier collected SO FAR was a keyword. This meant "WHERE Region" was merged into one token when tokenizing "SUM(Revenue WHERE Region = East)". Now the tokenizer also checks if the identifier built up to the current space IS a keyword (WHERE, SUM, AVG, MIN, MAX, COUNT, IF), which correctly produces separate tokens for "Revenue", "WHERE", "Region". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>