com.singularsys.jep.configurableparser
Class Tokenizer

java.lang.Object
  com.singularsys.jep.configurableparser.Tokenizer

public class Tokenizer
extends java.lang.Object

Breaks the input into a set of tokens.

Field Summary

protected java.io.BufferedReader	br
protected java.lang.String	currentLine
protected int	currentLineNumber
protected int	currentPos
static TerminatorToken	EOF A token which represents the end of file/input.
static TerminatorToken	EOL A token which represents and end of line.
protected java.util.List<TokenMatcher>	matchers
protected java.util.List<Token>	tokens

Constructor Summary

Tokenizer(java.io.BufferedReader br, java.util.List<TokenMatcher> matchers)
Construct a tokenizer for a given input.

Method Summary

java.util.List<Token>	scan() Scans the input, breaking it down into tokens.
static java.lang.String	toString(java.util.List<Token> toks)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

matchers

protected java.util.List<TokenMatcher> matchers

currentPos

protected int currentPos

currentLine

protected java.lang.String currentLine

currentLineNumber

protected int currentLineNumber

br

protected java.io.BufferedReader br

tokens

protected java.util.List<Token> tokens

EOL

public static TerminatorToken EOL

A token which represents and end of line.

EOF

public static TerminatorToken EOF

A token which represents the end of file/input. A TokenMatcher can return this symbol if it wishes to terminate processing of the input.

Constructor Detail

Tokenizer

public Tokenizer(java.io.BufferedReader br,
                 java.util.List<TokenMatcher> matchers)

Construct a tokenizer for a given input.

Parameters:

br - the input to read from.

matchers - a set of TokenMatchers to match the given input.

Method Detail

scan

public java.util.List<Token> scan()
                           throws ParseException

Scans the input, breaking it down into tokens. Returns null if attempts to read past end of the file

Returns:

the list of tokens that the input was broken down into

Throws:

ParseException

toString

public static java.lang.String toString(java.util.List<Token> toks)