Package com.singularsys.jep.configurableparser

# Class Tokenizer

java.lang.Object
com.singularsys.jep.configurableparser.Tokenizer

public class Tokenizer extends Object
Breaks the input into a set of tokens.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `Tokenizer.StandardTokenizerFactory`
  Factory creating new instances of the tokenizer.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected final BufferedReader`
  `br`
  `protected String`
  `currentLine`
  `protected int`
  `currentLineNumber`
  `protected int`
  `currentPos`
  `protected final List<TokenMatcher>`
  `matchers`
  `protected List<Token>`
  `tokens`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Tokenizer(BufferedReader br, ConfigurableParser cp)`
  Construct a tokenizer for a given input.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `List<Token>`
  `scan()`
  Scans the input, breaking it down into tokens.
  `static String`
  `toString(List<Token> toks)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### matchers

    protected final List<TokenMatcher> matchers

  - ### currentPos

    protected int currentPos

  - ### currentLine

    protected String currentLine

  - ### currentLineNumber

    protected int currentLineNumber

  - ### br

    protected final BufferedReader br

  - ### tokens

    protected List<Token> tokens

- ## Constructor Details

  - ### Tokenizer

    public Tokenizer(BufferedReader br, ConfigurableParser cp)
    Construct a tokenizer for a given input.
    Parameters:
      `br` - the input to read from.
      `cp` - The configurable parser instance.
    Since:
      3.4.0

- ## Method Details

  - ### scan

    public List<Token> scan() throws ParseException
    Scans the input, breaking it down into tokens.
    Returns:
      the list of tokens that the input was broken down into, ";;" returns an empty list, null if attempts to read past end of the file
    Throws:
      `ParseException` - if there is a syntax error in the input

  - ### toString

    public static String toString(List<Token> toks)