Tokenizer (original) (raw)

• java.lang.Object

• • weka.core.tokenizers.Tokenizer

• All Implemented Interfaces:
  java.io.Serializable, java.util.Enumeration, OptionHandler, RevisionHandler
  Direct Known Subclasses:
  AlphabeticTokenizer, CharacterDelimitedTokenizer

public abstract class Tokenizer
extends java.lang.Object
implements java.util.Enumeration, OptionHandler, java.io.Serializable, RevisionHandler
A superclass for all tokenizer algorithms.
Version: Revision:1.3Revision: 1.3 Revision:1.3
Author:
FracPete (fracpete at waikato dot ac dot nz)
See Also:
Serialized Form

• • Constructor Summary

  Constructors

  Constructor and Description
  Tokenizer()

  • Method Summary

  All Methods Static Methods Instance Methods Abstract Methods Concrete Methods

  Modifier and Type	Method and Description
  java.lang.String[]	getOptions() Gets the current option settings for the OptionHandler.
  abstract java.lang.String	globalInfo() Returns a string describing the stemmer
  abstract boolean	hasMoreElements() Tests if this enumeration contains more elements.
  java.util.Enumeration	listOptions() Returns an enumeration of all the available options..
  abstract java.lang.Object	nextElement() Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
  static void	runTokenizer(Tokenizer tokenizer, java.lang.String[] options) initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.
  void	setOptions(java.lang.String[] options) Sets the OptionHandler's options using the given list.
  abstract void	tokenize(java.lang.String s) Sets the string to tokenize.
  static java.lang.String[]	tokenize(Tokenizer tokenizer, java.lang.String[] options) initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.

     * ### Methods inherited from class java.lang.Object  
     `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`  
     * ### Methods inherited from interface weka.core.[RevisionHandler](../../../weka/core/RevisionHandler.html "interface in weka.core")  
     `[getRevision](../../../weka/core/RevisionHandler.html#getRevision--)`

• • Constructor Detail

     * #### Tokenizer  
     public Tokenizer()  

  • Method Detail

    * #### globalInfo  
    public abstract java.lang.String globalInfo()  
    Returns a string describing the stemmer  
    Returns:  
    a description suitable for displaying in the explorer/experimenter gui  
    * #### listOptions  
    public java.util.Enumeration listOptions()  
    Returns an enumeration of all the available options..  
    Specified by:  
    `[listOptions](../../../weka/core/OptionHandler.html#listOptions--)` in interface `[OptionHandler](../../../weka/core/OptionHandler.html "interface in weka.core")`  
    Returns:  
    an enumeration of all available options.  
    * #### getOptions  
    public java.lang.String[] getOptions()  
    Gets the current option settings for the OptionHandler.  
    Specified by:  
    `[getOptions](../../../weka/core/OptionHandler.html#getOptions--)` in interface `[OptionHandler](../../../weka/core/OptionHandler.html "interface in weka.core")`  
    Returns:  
    the list of current option settings as an array of strings  
    * #### setOptions  
    public void setOptions(java.lang.String[] options)  
                    throws java.lang.Exception  
    Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).  
    Specified by:  
    `[setOptions](../../../weka/core/OptionHandler.html#setOptions-java.lang.String:A-)` in interface `[OptionHandler](../../../weka/core/OptionHandler.html "interface in weka.core")`  
    Parameters:  
    `options` \- the list of options as an array of strings  
    Throws:  
    `java.lang.Exception` \- if an option is not supported  
    * #### hasMoreElements  
    public abstract boolean hasMoreElements()  
    Tests if this enumeration contains more elements.  
    Specified by:  
    `hasMoreElements` in interface `java.util.Enumeration`  
    Returns:  
    true if and only if this enumeration object contains at least one more element to provide; false otherwise.  
    * #### nextElement  
    public abstract java.lang.Object nextElement()  
    Returns the next element of this enumeration if this enumeration object has at least one more element to provide.  
    Specified by:  
    `nextElement` in interface `java.util.Enumeration`  
    Returns:  
    the next element of this enumeration.  
    * #### tokenize  
    public abstract void tokenize(java.lang.String s)  
    Sets the string to tokenize. Tokenization happens immediately.  
    Parameters:  
    `s` \- the string to tokenize  
    * #### tokenize  
    public static java.lang.String[] tokenize([Tokenizer](../../../weka/core/tokenizers/Tokenizer.html "class in weka.core.tokenizers") tokenizer,  
                                              java.lang.String[] options)  
                                       throws java.lang.Exception  
    initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. If no strings remained in the option string then data is read from stdin, line by line.  
    Parameters:  
    `tokenizer` \- the tokenizer to use  
    `options` \- the options for the tokenizer  
    Returns:  
    the tokenized strings  
    Throws:  
    `java.lang.Exception` \- if setting of options or tokenization fails  
    * #### runTokenizer  
    public static void runTokenizer([Tokenizer](../../../weka/core/tokenizers/Tokenizer.html "class in weka.core.tokenizers") tokenizer,  
                                    java.lang.String[] options)  
    initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. The generated tokens are then printed to stdout. If no strings remained in the option string then data is read from stdin, line by line.  
    Parameters:  
    `tokenizer` \- the tokenizer to use  
    `options` \- the options for the tokenizer