public class Tokenizer
extends java.lang.Object
Modifier and Type | Class and Description |
---|---|
static class |
Tokenizer.StandardTokenizerFactory
Factory creating new instances of the tokenizer.
|
Modifier and Type | Field and Description |
---|---|
protected java.io.BufferedReader |
br |
protected java.lang.String |
currentLine |
protected int |
currentLineNumber |
protected int |
currentPos |
protected java.util.List<TokenMatcher> |
matchers |
protected java.util.List<Token> |
tokens |
Constructor and Description |
---|
Tokenizer(java.io.BufferedReader br,
ConfigurableParser cp)
Construct a tokenizer for a given input.
|
Modifier and Type | Method and Description |
---|---|
java.util.List<Token> |
scan()
Scans the input, breaking it down into tokens.
|
static java.lang.String |
toString(java.util.List<Token> toks) |
protected java.util.List<TokenMatcher> matchers
protected int currentPos
protected java.lang.String currentLine
protected int currentLineNumber
protected java.io.BufferedReader br
protected java.util.List<Token> tokens
public Tokenizer(java.io.BufferedReader br, ConfigurableParser cp)
br
- the input to read from.cp
- The configurable parser instance.public java.util.List<Token> scan() throws ParseException
ParseException
- if there is a syntax error in the inputpublic static java.lang.String toString(java.util.List<Token> toks)
Copyright © 2018 Singular Systems http://www.singularsys.com/jep