Java StringTokenizer Class (original) (raw)

Last Updated : 10 Jan, 2025

**StringTokenizer class in Java is used to break a string into tokens based on delimiters. A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.

• A token is returned by taking a substring of the string that was used to create the StringTokenizer object.
• It provides the first step in the parsing process often called **lexer or scanner.
• It implements the Enumeration interface****.**
• To perform **Java String Tokenization, we need to specify an input string and a set of delimiters.
• A delimiter is a character or set of characters that separate tokens in the string.

**Note: StringTokenizer is a legacy class, and the split() method is preferred for modern applications.

**Example: Below is a simple example that explains the use of **Java StringTokenizer_to split a space-separated string into tokens:

Java `

// Demonstration of Java StringTokenizer import java.util.StringTokenizer;

public class Geeks { public static void main(String[] args) {

    // Input string
    String s = "Hello Geeks how are you";

    // Create a StringTokenizer object 
    // with space as the delimiter
    StringTokenizer st = new StringTokenizer(s, " ");

    // Tokenize the string and print each token
    while (st.hasMoreTokens()) {
        System.out.println(st.nextToken());
    }
}

}

`

Output

Hello Geeks how are you

**Explanation: In the above example, we have created a StringTokenizer object by passing the string and a space ” ” as the delimiter. The hasMoreTokens() method checks there are more tokens available to process or not. The nextToken() method get the next token (substring).

Below is the representation of the process, which we defined in the above example:

Constructors of StringTokenizer Class

The StringTokenizer class provides three constructors to tokenize strings in different ways.

**Note:

• **Default Delimiters: When no delimiter is specified, whitespace is used.
• **returnDelims: If true, the delimiters are treated as tokens themselves.

Below is a concise explanation of how each constructor works, along with a code example in the combined way.

Cases of StringTokenizer Constructors

**1. If the **returnDelims is false, delimiter characters serve to separate tokens.

**Example:

**Input: if string –> “hello geeks” and Delimiter is ” “, then
**Output: tokens are “hello” and “geeks”.

**2. If the **returnDelims is true, delimiter characters are considered to be tokens.

**Example:

**Input: String –> is “hello geeks” and Delimiter is ” “, then
**Output: Tokens –> “hello”, ” ” and “geeks”.

**3. Multiple delimiters can be chosen for a single string.

**Example:

**Syntax: StringTokenizer st1 = new StringTokenizer( “2+3-1*8/4”, “+*-/”);

**Input: String –> is “2+3-1*8/4” and Delimiters are +,*,-,/
**Output: Tokens –> “2”,”3″,”1″,”8″,”4″.

**Example:

Java `

// Demonstration of String Tokenizer Constructors import java.util.*;

class Geeks {

public static void main(String[] args) {
  
    // Example with Constructor 1
    System.out.println("Using StringTokenizer Constructor 1: ");

    // Using StringTokenizer to split the string into 
    // tokens using space (" ") as the delimiter
    StringTokenizer st1 = new StringTokenizer(
        "Geeks fo Geeks", " ");

    // Iterate through tokens while 
    // there are more tokens available
    while (st1.hasMoreTokens())
      
        // Getting and printing the next token
        System.out.println(st1.nextToken());

    // Example with Constructor 2
    System.out.println("Using StringTokenizer Constructor 2: ");

    // Using StringTokenizer to split the string 
    // using ":" as the delimiter
    StringTokenizer st2 = new StringTokenizer(
        "java : Code : String : Tokenizer", " :");

    // Iterate through tokens and print them
    while (st2.hasMoreTokens())
        System.out.println(st2.nextToken());

    // Example with Constructor 3
    System.out.println("Using StringTokenizer Constructor 3: ");

    // Using StringTokenizer with returnDelims = true 
    // to include delimiters as tokens
    StringTokenizer st3 = new StringTokenizer(
        "java : Code", " :", true);

    // Iterate through tokens (including delimiters) 
    // and print them
    while (st3.hasMoreTokens())
        System.out.println(st3.nextToken());
}

}

`

Output

Using StringTokenizer Constructor 1: Geeks fo Geeks Using StringTokenizer Constructor 2: java Code String Tokenizer Using StringTokenizer Constructor 3: java

:

Code

Methods Of StringTokenizer Class

Below are some commonly used methods of StringTokenizer class along with a combined code example demonstrating some of these methods.

**Example:

Java `

// Demonstration of StringTokenizer Methods import java.util.*;

class Geeks { public static void main(String[] args) {

    // Creating a StringTokenizer
    StringTokenizer st = new StringTokenizer(
        "Welcome to GeeksforGeeks");

    StringTokenizer st1 = new StringTokenizer("");

      // countTokens Method
    int c = st.countTokens();
    System.out.println(c);
  
      // hasMoreTokens Methods
      System.out.println("Welcome to GeeksforGeeks: "+ st.hasMoreTokens());
      System.out.println("(Empty String) : "+ st1.hasMoreTokens());
  
      // nextElement() Method
      System.out.println("\nTraversing the String:");
  
      while(st.hasMoreTokens()){
          System.out.println(st.nextElement());
    }
      
}

}

`

Output

3 Welcome to GeeksforGeeks: true (Empty String) : false

Traversing the String: Welcome to GeeksforGeeks