Package org.pipservices3.expressions.tokenizers.generic

Class GenericWordState

java.lang.Object
org.pipservices3.expressions.tokenizers.generic.GenericWordState
All Implemented Interfaces:
ITokenizerState, IWordState
Direct Known Subclasses:
CsvWordState, ExpressionWordState
public class GenericWordState extends Object implements IWordState
A wordState returns a word from a scanner. Like other states, a tokenizer transfers the job of reading to this state, depending on an initial character. Thus, the tokenizer decides which characters may begin a word, and this state determines which characters may appear as a second or later character in a word. These are typically different sets of characters; in particular, it is typical for digits to appear as parts of a word, but not as the initial character of a word.

By default, the following characters may appear in a word. The method setWordChars() allows customizing this. 

 From    To
   'a', 'z'
   'A', 'Z'
   '0', '9'

    as well as: minus sign, underscore, and apostrophe.

  • Constructor Summary

    Constructors
    Constructor
    Description
    GenericWordState()
    Constructs a word state with a default idea of what characters are admissible inside a word (as described in the class comment).

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    clearWordChars()
    Clears definitions of word chars.
    Token
    nextToken(IScanner scanner, ITokenizer tokenizer)
    Ignore word (such as blanks and tabs), and return the tokenizer's next token.
    void
    setWordChars(int fromSymbol, int toSymbol, boolean enable)
    Establish characters in the given range as valid characters for part of a word after the first character.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • GenericWordState

      public GenericWordState() throws Exception
      Constructs a word state with a default idea of what characters are admissible inside a word (as described in the class comment).
      Throws:
      Exception

  • Method Details

    • nextToken

      public Token nextToken(IScanner scanner, ITokenizer tokenizer) throws Exception
      Ignore word (such as blanks and tabs), and return the tokenizer's next token.
      Specified by:
      nextToken in interface ITokenizerState
      Parameters:
      scanner - A textual string to be tokenized.
      tokenizer - A tokenizer class that controls the process.
      Returns:
      The next token from the top of the stream.
      Throws:
      Exception

    • setWordChars

      public void setWordChars(int fromSymbol, int toSymbol, boolean enable) throws Exception
      Establish characters in the given range as valid characters for part of a word after the first character. Note that the tokenizer must determine which characters are valid as the beginning character of a word.
      Specified by:
      setWordChars in interface IWordState
      Parameters:
      fromSymbol - First character index of the interval.
      toSymbol - Last character index of the interval.
      enable - true if this state should use characters in the given range.
      Throws:
      Exception

    • clearWordChars

      public void clearWordChars()
      Clears definitions of word chars.
      Specified by:
      clearWordChars in interface IWordState