The Tokenizer type exposes the following members.
Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection.(Inherited from Object.)
Serves as a hash function for a particular type.(Inherited from Object.)
Gets the Type of the current instance.(Inherited from Object.)
Creates a shallow copy of the current Object.(Inherited from Object.)
Scans the input, breaking it down into tokens. Returns null if attempts to read past end of the file