Package org.pipservices3.expressions.tokenizers

Interface ITokenizerState

All Known Subinterfaces:
ICommentState, INumberState, IQuoteState, ISymbolState, IWhitespaceState, IWordState
All Known Implementing Classes:
CCommentState, CppCommentState, CsvQuoteState, CsvSymbolState, CsvWordState, ExpressionNumberState, ExpressionQuoteState, ExpressionSymbolState, ExpressionWordState, GenericCommentState, GenericNumberState, GenericQuoteState, GenericSymbolState, GenericWhitespaceState, GenericWordState, MustacheSpecialState
public interface ITokenizerState
A tokenizerState returns a token, given a scanner, an initial character read from the scanner, and a tokenizer that is conducting an overall tokenization of the scanner. The tokenizer will typically have a character state table that decides which state to use, depending on an initial character. If a single character is insufficient, a state such as SlashState will read a second character, and may delegate to another state, such as SlashStarState. This prospect of delegation is the reason that the nextToken() method has a tokenizer argument.

  • Method Summary

    Modifier and Type
    Method
    Description
    Token
    nextToken(IScanner scanner, ITokenizer tokenizer)
    Gets the next token from the stream started from the character linked to this state.

  • Method Details

    • nextToken

      Token nextToken(IScanner scanner, ITokenizer tokenizer) throws Exception
      Gets the next token from the stream started from the character linked to this state.
      Parameters:
      scanner - A textual string to be tokenized.
      tokenizer - A tokenizer class that controls the process.
      Returns:
      The next token from the top of the stream.
      Throws:
      Exception