TokenRecognizer

A token recognizer has the ability to detect a pattern within a string (the exact pattern being entirely up to the recognizer) and then returns the matched pattern as well as the ending index (that is, the index of the last matched character + 1, this is considered to be an exclusive index).

Recognizers cannot be used as-is within a lexer, as lexers expect matchers. To use a recognizer within a lexer, use the isToken method inside a state block for a DSL approach, or a TokenRecognizerMatched object.

Inheritors