org.simmetrics.tokenizers

Interface Tokenizer

All Known Subinterfaces:

TokenizingTokenizer

All Known Implementing Classes:

AbstractTokenizer, CachingTokenizer, QGram, QGramExtended, Whitespace

public interface Tokenizer

A tokenizer divides an input string into tokens. A tokenizer may not provide null as a token.

Method Summary

Modifier and Type	Method and Description
List<String>	tokenizeToList(String input) Return tokenized version of a string as a list of tokens.
Set<String>	tokenizeToSet(String input) Return tokenized version of a string as a set of tokens.

Method Detail

tokenizeToList

List<String> tokenizeToList(String input)

Return tokenized version of a string as a list of tokens.

Parameters:

input - input string to tokenize

Returns:

List tokenized version of a string

tokenizeToSet

Set<String> tokenizeToSet(String input)

Return tokenized version of a string as a set of tokens.

Parameters:

input - input string to tokenize

Returns:

tokenized version of a string as a set