org.simmetrics.tokenizers

Class AbstractTokenizer

java.lang.Object

org.simmetrics.tokenizers.AbstractTokenizer

All Implemented Interfaces:

Tokenizer

Direct Known Subclasses:

QGram, QGramExtended, Whitespace

public abstract class AbstractTokenizer
extends Object
implements Tokenizer

Convenience tokenizer. Provides default implementation to tokenize to set and multi set.

Constructor Summary

Constructors

Constructor and Description
AbstractTokenizer()

Method Summary

Modifier and Type	Method and Description
Set<String>	tokenizeToSet(String input) Return tokenized version of a string as a set of tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.simmetrics.tokenizers.Tokenizer

tokenizeToList

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()

Method Detail

tokenizeToSet

public Set<String> tokenizeToSet(String input)

Description copied from interface: Tokenizer

Return tokenized version of a string as a set of tokens.

Specified by:

tokenizeToSet in interface Tokenizer

Parameters:

input - input string to tokenize

Returns:

tokenized version of a string as a set