Package org.pipservices3.expressions.tokenizers

Interface ITokenizer

All Known Implementing Classes:
AbstractTokenizer, CsvTokenizer, ExpressionTokenizer, GenericTokenizer, MustacheTokenizer
public interface ITokenizer

  • Method Details

    • getSkipUnknown

      Boolean getSkipUnknown()
      Skip unknown characters

    • setSkipUnknown

      void setSkipUnknown(boolean value)
      Skip unknown characters

    • getSkipWhitespaces

      Boolean getSkipWhitespaces()
      Skips whitespaces.

    • setSkipWhitespaces

      void setSkipWhitespaces(boolean value)
      Skips whitespaces.

    • getSkipComments

      Boolean getSkipComments()
      Skips comments.

    • setSkipComments

      void setSkipComments(boolean value)
      Skips comments.

    • getSkipEof

      Boolean getSkipEof()
      Skips End-Of-File token at the end of stream.

    • setSkipEof

      void setSkipEof(boolean value)
      Skips End-Of-File token at the end of stream.

    • getMergeWhitespaces

      Boolean getMergeWhitespaces()
      Merges whitespaces.

    • setMergeWhitespaces

      void setMergeWhitespaces(boolean value)
      Merges whitespaces.

    • getUnifyNumbers

      Boolean getUnifyNumbers()
      Unifies numbers: "Integers" and "Floats" makes just "Numbers"

    • setUnifyNumbers

      void setUnifyNumbers(boolean value)
      Unifies numbers: "Integers" and "Floats" makes just "Numbers"

    • getDecodeStrings

      Boolean getDecodeStrings()
      Decodes quoted strings.

    • setDecodeStrings

      void setDecodeStrings(boolean value)
      Decodes quoted strings.

    • getCommentState

      ICommentState getCommentState()
      A token state to process comments.

    • setCommentState

      void setCommentState(ICommentState value)
      A token state to process comments.

    • getNumberState

      INumberState getNumberState()
      A token state to process numbers.

    • setNumberState

      void setNumberState(INumberState value)
      A token state to process numbers.

    • getQuoteState

      IQuoteState getQuoteState()
      A token state to process quoted strings.

    • setQuoteState

      void setQuoteState(IQuoteState value)
      A token state to process quoted strings.

    • getSymbolState

      ISymbolState getSymbolState()
      A token state to process symbols (single like "=" or muti-character like "<>")

    • setSymbolState

      void setSymbolState(ISymbolState value)
      A token state to process symbols (single like "=" or muti-character like "<>")

    • getWhitespaceState

      IWhitespaceState getWhitespaceState()
      A token state to process white space delimiters.

    • setWhitespaceState

      void setWhitespaceState(IWhitespaceState value)
      A token state to process white space delimiters.

    • getWordState

      IWordState getWordState()
      A token state to process words or indentificators.

    • setWordState

      void setWordState(IWordState value)
      A token state to process words or indentificators.

    • getScanner

      IScanner getScanner()
      The stream scanner to tokenize.

    • setScanner

      void setScanner(IScanner scanner)
      The stream scanner to tokenize.

    • hasNextToken

      Boolean hasNextToken() throws Exception
      Checks if there is the next token exist.
      Returns:
      true if scanner has the next token.
      Throws:
      Exception

    • nextToken

      Token nextToken() throws Exception
      Gets the next token from the scanner.
      Returns:
      Next token of null if there are no more tokens left.
      Throws:
      Exception

    • tokenizeStream

      List<Token> tokenizeStream(IScanner scanner) throws Exception
      Tokenizes a textual stream into a list of token structures.
      Parameters:
      scanner - A textual stream to be tokenized.
      Returns:
      A list of token structures.
      Throws:
      Exception

    • tokenizeBuffer

      List<Token> tokenizeBuffer(String buffer) throws Exception
      Tokenizes a string buffer into a list of tokens structures.
      Parameters:
      buffer - A string buffer to be tokenized.
      Returns:
      A list of token structures.
      Throws:
      Exception

    • tokenizeStreamToStrings

      List<String> tokenizeStreamToStrings(IScanner scanner) throws Exception
      Tokenizes a textual stream into a list of strings.
      Parameters:
      scanner - A textual stream to be tokenized.
      Returns:
      A list of token strings.
      Throws:
      Exception

    • tokenizeBufferToStrings

      List<String> tokenizeBufferToStrings(String buffer) throws Exception
      Tokenizes a string buffer into a list of strings.
      Parameters:
      buffer - A string buffer to be tokenized.
      Returns:
      A list of token strings.
      Throws:
      Exception