UnicodeTokenizer (1.2 API)

java.lang.Object

de.l3s.boilerpipe.util.UnicodeTokenizer

public class UnicodeTokenizer extends Object

Tokenizes text according to Unicode word boundaries and strips off non-word characters.

Author:: Christian Kohlschütter

Constructor Summary

Constructors

Constructor

Description

UnicodeTokenizer()
Method Summary

Modifier and Type

Method

Description

static String[]

tokenize(CharSequence text)

Tokenizes the text and returns an array of tokens.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details
- UnicodeTokenizer
  
  public UnicodeTokenizer()
Method Details
- tokenize
  
  public static String[] tokenize(CharSequence text)
  
  Tokenizes the text and returns an array of tokens.
  
  Parameters:
  
  text - The text
  
  Returns:
  
  The tokens