Class Tokenizer

java.lang.Object
  extended byTokenizer

public class Tokenizer
extends java.lang.Object

Tokenizes input strings prior to parsing.


Constructor Summary
Tokenizer()
          Create a new Tokenizer.
 
Method Summary
 java.util.List tokenizeInput(java.lang.String sentence)
          Tokenize an input sentence and return a List of Symbols.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Tokenizer

public Tokenizer()
Create a new Tokenizer. For now, this is uses just a StringTokenizer, but eventually it could do some morphology, spelling correction, etc.

Method Detail

tokenizeInput

public java.util.List tokenizeInput(java.lang.String sentence)
Tokenize an input sentence and return a List of Symbols.