Tokenizer (Jep with extensions API )

java.lang.Object
- com.singularsys.jep.configurableparser.Tokenizer

public class Tokenizer
extends java.lang.Object

Breaks the input into a set of tokens.

Nested Class Summary

Nested Classes
Modifier and Type Class Description

static class Tokenizer.StandardTokenizerFactory
Factory creating new instances of the tokenizer.

Field Summary

Fields
Modifier and Type	Field	Description
`protected java.io.BufferedReader`	`br`
`protected java.lang.String`	`currentLine`
`protected int`	`currentLineNumber`
`protected int`	`currentPos`
`protected java.util.List<TokenMatcher>`	`matchers`
`protected java.util.List<Token>`	`tokens`

Constructor Summary

Constructors
Constructor Description

Tokenizer(java.io.BufferedReader br, ConfigurableParser cp)
Construct a tokenizer for a given input.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`java.util.List<Token>`	`scan()`	Scans the input, breaking it down into tokens.
`static java.lang.String`	`toString(java.util.List<Token> toks)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - matchers
```
protected java.util.List<TokenMatcher> matchers
```
  - currentPos
```
protected int currentPos
```
  - currentLine
```
protected java.lang.String currentLine
```
  - currentLineNumber
```
protected int currentLineNumber
```
  - br
```
protected java.io.BufferedReader br
```
  - tokens
```
protected java.util.List<Token> tokens
```
- Constructor Detail
  - Tokenizer
```
public Tokenizer(java.io.BufferedReader br,
                 ConfigurableParser cp)
```
    Construct a tokenizer for a given input.
    
    Parameters:
    
    br - the input to read from.
    
    cp - The configurable parser instance.
    
    Since:
    
    3.4.0
- Method Detail
  - scan
```
public java.util.List<Token> scan()
                           throws ParseException
```
    Scans the input, breaking it down into tokens.
    
    Returns:
    
    the list of tokens that the input was broken down into, ";;" returns an empty list, null if attempts to read past end of the file
    
    Throws:
    
    ParseException - if there is a syntax error in the input
  - toString
```
public static java.lang.String toString(java.util.List<Token> toks)
```