Package org.pipservices3.expressions.tokenizers
package org.pipservices3.expressions.tokenizers
-
ClassDescriptionImplements an abstract tokenizer class.Defines an interface for tokenizer state that processes comments.Defines interface for tokenizer state that processes numbers - Integers, Floats, HexDec..Defines an interface for tokenizer state that processes quoted strings.Defines an interface for tokenizer state that processes delimiters.A tokenizerState returns a token, given a scanner, an initial character read from the scanner, and a tokenizer that is conducting an overall tokenization of the scanner.Defines an interface for tokenizer state that processes whitespaces (' ', '\t')Defines an interface for tokenizer state that processes words, identificators or keywordsA token represents a logical chunk of a string.Types (categories) of tokens such as "number", "symbol" or "word".