Package org.pipservices3.expressions.tokenizers

Class AbstractTokenizer

java.lang.Object

org.pipservices3.expressions.tokenizers.AbstractTokenizer

All Implemented Interfaces:

ITokenizer

Direct Known Subclasses:

CsvTokenizer, ExpressionTokenizer, GenericTokenizer, MustacheTokenizer

public abstract class AbstractTokenizer
extends Object
implements ITokenizer

Implements an abstract tokenizer class.

Field Summary

Fields

Modifier and Type	Field	Description
protected TokenType	_lastTokenType
protected Token	_nextToken
protected IScanner	_scanner

Constructor Summary

Constructors

Constructor	Description
AbstractTokenizer()

Method Summary

Modifier and Type	Method	Description
void	clearCharacterStates()
ITokenizerState	getCharacterState(int symbol)
ICommentState	getCommentState()	A token state to process comments.
Boolean	getDecodeStrings()	Decodes quoted strings.
Boolean	getMergeWhitespaces()	Merges whitespaces.
INumberState	getNumberState()	A token state to process numbers.
IQuoteState	getQuoteState()	A token state to process quoted strings.
IScanner	getScanner()	The stream scanner to tokenize.
Boolean	getSkipComments()	Skips comments.
Boolean	getSkipEof()	Skips End-Of-File token at the end of stream.
Boolean	getSkipUnknown()	Skip unknown characters
Boolean	getSkipWhitespaces()	Skips whitespaces.
ISymbolState	getSymbolState()	A token state to process symbols (single like "=" or muti-character like "<>")
Boolean	getUnifyNumbers()	Unifies numbers: "Integers" and "Floats" makes just "Numbers"
IWhitespaceState	getWhitespaceState()	A token state to process white space delimiters.
IWordState	getWordState()	A token state to process words or indentificators.
Boolean	hasNextToken()	Checks if there is the next token exist.
Token	nextToken()	Gets the next token from the scanner.
protected Token	readNextToken()
void	setCharacterState(int fromSymbol, int toSymbol, ITokenizerState state)
void	setCommentState(ICommentState value)	A token state to process comments.
void	setDecodeStrings(boolean value)	Decodes quoted strings.
void	setMergeWhitespaces(boolean value)	Merges whitespaces.
void	setNumberState(INumberState value)	A token state to process numbers.
void	setQuoteState(IQuoteState value)	A token state to process quoted strings.
void	setScanner(IScanner value)	The stream scanner to tokenize.
void	setSkipComments(boolean value)	Skips comments.
void	setSkipEof(boolean value)	Skips End-Of-File token at the end of stream.
void	setSkipUnknown(boolean value)	Skip unknown characters
void	setSkipWhitespaces(boolean value)	Skips whitespaces.
void	setSymbolState(ISymbolState value)	A token state to process symbols (single like "=" or muti-character like "<>")
void	setUnifyNumbers(boolean value)	Unifies numbers: "Integers" and "Floats" makes just "Numbers"
void	setWhitespaceState(IWhitespaceState value)	A token state to process white space delimiters.
void	setWordState(IWordState value)	A token state to process words or indentificators.
List<Token>	tokenizeBuffer(String buffer)	Tokenizes a string buffer into a list of tokens structures.
List<String>	tokenizeBufferToStrings(String buffer)	Tokenizes a string buffer into a list of strings.
List<Token>	tokenizeStream(IScanner scanner)	Tokenizes a textual stream into a list of token structures.
List<String>	tokenizeStreamToStrings(IScanner scanner)	Tokenizes a textual stream into a list of strings.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

_scanner

protected IScanner _scanner

_nextToken

protected Token _nextToken

_lastTokenType

protected TokenType _lastTokenType

Constructor Details

AbstractTokenizer

public AbstractTokenizer()

Method Details

getSkipUnknown

public Boolean getSkipUnknown()

Description copied from interface: ITokenizer

Skip unknown characters

Specified by:

getSkipUnknown in interface ITokenizer

setSkipUnknown

public void setSkipUnknown(boolean value)

Description copied from interface: ITokenizer

Skip unknown characters

Specified by:

setSkipUnknown in interface ITokenizer

getSkipWhitespaces

public Boolean getSkipWhitespaces()

Description copied from interface: ITokenizer

Skips whitespaces.

Specified by:

getSkipWhitespaces in interface ITokenizer

setSkipWhitespaces

public void setSkipWhitespaces(boolean value)

Description copied from interface: ITokenizer

Skips whitespaces.

Specified by:

setSkipWhitespaces in interface ITokenizer

getSkipComments

public Boolean getSkipComments()

Description copied from interface: ITokenizer

Skips comments.

Specified by:

getSkipComments in interface ITokenizer

setSkipComments

public void setSkipComments(boolean value)

Description copied from interface: ITokenizer

Skips comments.

Specified by:

setSkipComments in interface ITokenizer

getSkipEof

public Boolean getSkipEof()

Description copied from interface: ITokenizer

Skips End-Of-File token at the end of stream.

Specified by:

getSkipEof in interface ITokenizer

setSkipEof

public void setSkipEof(boolean value)

Description copied from interface: ITokenizer

Skips End-Of-File token at the end of stream.

Specified by:

setSkipEof in interface ITokenizer

getMergeWhitespaces

public Boolean getMergeWhitespaces()

Description copied from interface: ITokenizer

Merges whitespaces.

Specified by:

getMergeWhitespaces in interface ITokenizer

setMergeWhitespaces

public void setMergeWhitespaces(boolean value)

Description copied from interface: ITokenizer

Merges whitespaces.

Specified by:

setMergeWhitespaces in interface ITokenizer

getUnifyNumbers

public Boolean getUnifyNumbers()

Description copied from interface: ITokenizer

Unifies numbers: "Integers" and "Floats" makes just "Numbers"

Specified by:

getUnifyNumbers in interface ITokenizer

setUnifyNumbers

public void setUnifyNumbers(boolean value)

Description copied from interface: ITokenizer

Unifies numbers: "Integers" and "Floats" makes just "Numbers"

Specified by:

setUnifyNumbers in interface ITokenizer

getDecodeStrings

public Boolean getDecodeStrings()

Description copied from interface: ITokenizer

Decodes quoted strings.

Specified by:

getDecodeStrings in interface ITokenizer

setDecodeStrings

public void setDecodeStrings(boolean value)

Description copied from interface: ITokenizer

Decodes quoted strings.

Specified by:

setDecodeStrings in interface ITokenizer

getCommentState

public ICommentState getCommentState()

Description copied from interface: ITokenizer

A token state to process comments.

Specified by:

getCommentState in interface ITokenizer

setCommentState

public void setCommentState(ICommentState value)

Description copied from interface: ITokenizer

A token state to process comments.

Specified by:

setCommentState in interface ITokenizer

getNumberState

public INumberState getNumberState()

Description copied from interface: ITokenizer

A token state to process numbers.

Specified by:

getNumberState in interface ITokenizer

setNumberState

public void setNumberState(INumberState value)

Description copied from interface: ITokenizer

A token state to process numbers.

Specified by:

setNumberState in interface ITokenizer

getQuoteState

public IQuoteState getQuoteState()

Description copied from interface: ITokenizer

A token state to process quoted strings.

Specified by:

getQuoteState in interface ITokenizer

setQuoteState

public void setQuoteState(IQuoteState value)

Description copied from interface: ITokenizer

A token state to process quoted strings.

Specified by:

setQuoteState in interface ITokenizer

getSymbolState

public ISymbolState getSymbolState()

Description copied from interface: ITokenizer

A token state to process symbols (single like "=" or muti-character like "<>")

Specified by:

getSymbolState in interface ITokenizer

setSymbolState

public void setSymbolState(ISymbolState value)

Description copied from interface: ITokenizer

A token state to process symbols (single like "=" or muti-character like "<>")

Specified by:

setSymbolState in interface ITokenizer

getWhitespaceState

public IWhitespaceState getWhitespaceState()

Description copied from interface: ITokenizer

A token state to process white space delimiters.

Specified by:

getWhitespaceState in interface ITokenizer

setWhitespaceState

public void setWhitespaceState(IWhitespaceState value)

Description copied from interface: ITokenizer

A token state to process white space delimiters.

Specified by:

setWhitespaceState in interface ITokenizer

getWordState

public IWordState getWordState()

Description copied from interface: ITokenizer

A token state to process words or indentificators.

Specified by:

getWordState in interface ITokenizer

setWordState

public void setWordState(IWordState value)

Description copied from interface: ITokenizer

A token state to process words or indentificators.

Specified by:

setWordState in interface ITokenizer

getScanner

public IScanner getScanner()

Description copied from interface: ITokenizer

The stream scanner to tokenize.

Specified by:

getScanner in interface ITokenizer

setScanner

public void setScanner(IScanner value)

Description copied from interface: ITokenizer

The stream scanner to tokenize.

Specified by:

setScanner in interface ITokenizer

getCharacterState

public ITokenizerState getCharacterState(int symbol)

setCharacterState

public void setCharacterState(int fromSymbol,
 int toSymbol,
 ITokenizerState state)
                       throws Exception

Throws:

Exception

clearCharacterStates

public void clearCharacterStates()

hasNextToken

public Boolean hasNextToken()
                     throws Exception

Description copied from interface: ITokenizer

Checks if there is the next token exist.

Specified by:

hasNextToken in interface ITokenizer

Returns:

true if scanner has the next token.

Throws:

Exception

nextToken

public Token nextToken()
                throws Exception

Description copied from interface: ITokenizer

Gets the next token from the scanner.

Specified by:

nextToken in interface ITokenizer

Returns:

Next token of null if there are no more tokens left.

Throws:

Exception

readNextToken

protected Token readNextToken()
                       throws Exception

Throws:

Exception

tokenizeStream

public List<Token> tokenizeStream(IScanner scanner)
                           throws Exception

Description copied from interface: ITokenizer

Tokenizes a textual stream into a list of token structures.

Specified by:

tokenizeStream in interface ITokenizer

Parameters:

scanner - A textual stream to be tokenized.

Returns:

A list of token structures.

Throws:

Exception

tokenizeBuffer

public List<Token> tokenizeBuffer(String buffer)
                           throws Exception

Description copied from interface: ITokenizer

Tokenizes a string buffer into a list of tokens structures.

Specified by:

tokenizeBuffer in interface ITokenizer

Parameters:

buffer - A string buffer to be tokenized.

Returns:

A list of token structures.

Throws:

Exception

tokenizeStreamToStrings

public List<String> tokenizeStreamToStrings(IScanner scanner)
                                     throws Exception

Description copied from interface: ITokenizer

Tokenizes a textual stream into a list of strings.

Specified by:

tokenizeStreamToStrings in interface ITokenizer

Parameters:

scanner - A textual stream to be tokenized.

Returns:

A list of token strings.

Throws:

Exception

tokenizeBufferToStrings

public List<String> tokenizeBufferToStrings(String buffer)
                                     throws Exception

Description copied from interface: ITokenizer

Tokenizes a string buffer into a list of strings.

Specified by:

tokenizeBufferToStrings in interface ITokenizer

Parameters:

buffer - A string buffer to be tokenized.

Returns:

A list of token strings.

Throws:

Exception