The Tokenizer type exposes the following members.
Constructors
Name | Description | |
---|---|---|
Tokenizer |
Construct a tokenizer for a given input.
|
Methods
Name | Description | |
---|---|---|
Equals | (Inherited from Object.) | |
Finalize |
Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection.
(Inherited from Object.) |
|
GetHashCode |
Serves as a hash function for a particular type.
(Inherited from Object.) |
|
GetType |
Gets the Type of the current instance.
(Inherited from Object.) |
|
MemberwiseClone |
Creates a shallow copy of the current Object.
(Inherited from Object.) |
|
Scan |
Scans the input, breaking it down into tokens.
Returns null if attempts to read past end of the file
|
|
ToString | Overloaded. |
Fields
Name | Description | |
---|---|---|
currentLine | ||
currentLineNumber | ||
currentPos | ||
EOF |
A token which represents the end of file/input.
A TokenMatcher can return this symbol if it wishes to terminate processing of the input.
|
|
EOL |
A token which represents and end of line.
|
|
matchers | ||
sr | ||
tokens |