com.singularsys.jep.configurableparser
Class Tokenizer

java.lang.Object
  com.singularsys.jep.configurableparser.Tokenizer

public class Tokenizer
extends java.lang.Object

Breaks the input into a set of tokens.

Nested Class Summary

static class

Tokenizer.StandardTokenizerFactory Factory creating new instances of the tokenizer.

Field Summary

protected java.io.BufferedReader	br
protected java.lang.String	currentLine
protected int	currentLineNumber
protected int	currentPos
protected java.util.List<TokenMatcher>	matchers
protected java.util.List<Token>	tokens

Constructor Summary

Tokenizer(java.io.BufferedReader br, ConfigurableParser cp)
Construct a tokenizer for a given input.

Method Summary

java.util.List<Token>	scan() Scans the input, breaking it down into tokens.
static java.lang.String	toString(java.util.List<Token> toks)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

matchers

protected java.util.List<TokenMatcher> matchers

currentPos

protected int currentPos

currentLine

protected java.lang.String currentLine

currentLineNumber

protected int currentLineNumber

br

protected java.io.BufferedReader br

tokens

protected java.util.List<Token> tokens

Constructor Detail

Tokenizer

public Tokenizer(java.io.BufferedReader br,
                 ConfigurableParser cp)

Construct a tokenizer for a given input.

Parameters:

br - the input to read from.

cp - The configurable parser instance.

Since:

3.4.0

Method Detail

scan

public java.util.List<Token> scan()
                           throws ParseException

Scans the input, breaking it down into tokens.

Returns:

the list of tokens that the input was broken down into, ";;" returns an empty list, null if attempts to read past end of the file

Throws:

ParseException

toString

public static java.lang.String toString(java.util.List<Token> toks)