Tokenizers (SimMetrics 3.2.3 API)

java.lang.Object
- org.simmetrics.tokenizers.Tokenizers

```
public final class Tokenizers
extends Object
```
Utilities for tokenizers. Construct simple tokenizers, chain multiple tokenizers into a single tokenizers or apply filters and transforms to tokens.
All methods return immutable objects provided the arguments are also immutable.

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method and Description
`static Tokenizer`	`chain(List<Tokenizer> tokenizers)` Chains tokenizers together.
`static Tokenizer`	`chain(Tokenizer tokenizer, Tokenizer... tokenizers)` Chains tokenizers together.
`static Tokenizer`	`filter(Tokenizer tokenizer, com.google.common.base.Predicate<String> predicate)` Constructs a new filtering tokenizer.
`static Tokenizer`	`pattern(Pattern pattern)` Returns a tokenizer that splits a string into tokens around the pattern as if calling `pattern.split(input,-1)`.
`static Tokenizer`	`pattern(String regex)` Returns a tokenizer that splits a string into tokens around the pattern as if calling `Pattern.compile(regex).split(input,-1)`.
`static Tokenizer`	`qGram(int q)` Returns a basic q-gram tokenizer for a variable q.
`static Tokenizer`	`qGramWithFilter(int q)` Returns a basic q-gram tokenizer for a variable q.
`static Tokenizer`	`qGramWithPadding(int q)` Returns a basic q-gram tokenizer for a variable q.The input is padded with q-1 special characters before being tokenized.
`static Tokenizer`	`qGramWithPadding(int q, String padding)` Returns a basic q-gram tokenizer for a variable Q.The Q-Gram is extended beyond the length of the string with padding.
`static Tokenizer`	`qGramWithPadding(int q, String startPadding, String endPadding)` Returns a basic q-gram tokenizer for a variable Q.The Q-Gram is extended beyond the length of the string with padding.
`static Tokenizer`	`transform(Tokenizer tokenizer, com.google.common.base.Function<String,String> function)` Constructs a new transforming tokenizer.
`static Tokenizer`	`whitespace()` Returns a tokenizer that splits a string into tokens around whitespace.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - chain
```
public static Tokenizer chain(List<Tokenizer> tokenizers)
```
    Chains tokenizers together. The output of each tokenizer is tokenized by the next. The tokenizers are applied in order. If only a single tokenizer is provided, that tokenizer is returned.
    
    Parameters:
    
    tokenizers - a non-empty list of tokenizers
    
    Returns:
    
    a chain of tokenizers
  - chain
```
public static Tokenizer chain(Tokenizer tokenizer,
                              Tokenizer... tokenizers)
```
    Chains tokenizers together. The output of each tokenizer is tokenized by the next. The tokenizers are applied in order. If only a single tokenizer is provided, that tokenizer is returned.
    
    Parameters:
    
    tokenizer - the first tokenizer
    
    tokenizers - a the other tokenizers
    
    Returns:
    
    a chain of tokenizers
  - filter
```
public static Tokenizer filter(Tokenizer tokenizer,
                               com.google.common.base.Predicate<String> predicate)
```
    Constructs a new filtering tokenizer. After tokenization, all tokens that don't match a predicate are removed.
    
    Parameters:
    
    tokenizer - delegate tokenizer
    
    predicate - for tokens to keep
    
    Returns:
    
    a new filtering tokenizer.
  - pattern
```
public static Tokenizer pattern(Pattern pattern)
```
    Returns a tokenizer that splits a string into tokens around the pattern as if calling pattern.split(input,-1).
    
    Parameters:
    
    pattern - to split the the string around
    
    Returns:
    
    a pattern tokenizer
  - pattern
```
public static Tokenizer pattern(String regex)
```
    Returns a tokenizer that splits a string into tokens around the pattern as if calling Pattern.compile(regex).split(input,-1).
    
    Parameters:
    
    regex - to split the the string around
    
    Returns:
    
    a pattern tokenizer
  - qGram
```
public static Tokenizer qGram(int q)
```
    Returns a basic q-gram tokenizer for a variable q. The tokenizer will return an empty list if the input is empty. A list with the original input is returned for tokens shorter then q.
    
    Parameters:
    
    q - size of the tokens
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithFilter
```
public static Tokenizer qGramWithFilter(int q)
```
    Returns a basic q-gram tokenizer for a variable q. The tokenizer will return an empty list if the input is empty or shorter then q.
    
    Parameters:
    
    q - size of the tokens
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithPadding
```
public static Tokenizer qGramWithPadding(int q)
```
    Returns a basic q-gram tokenizer for a variable q.The input is padded with q-1 special characters before being tokenized. Uses # as the default padding.
    
    Parameters:
    
    q - size of the tokens
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithPadding
```
public static Tokenizer qGramWithPadding(int q,
                                         String padding)
```
    Returns a basic q-gram tokenizer for a variable Q.The Q-Gram is extended beyond the length of the string with padding.
    
    Parameters:
    
    q - size of the tokens
    
    padding - padding to padd start and end of string with
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithPadding
```
public static Tokenizer qGramWithPadding(int q,
                                         String startPadding,
                                         String endPadding)
```
    Returns a basic q-gram tokenizer for a variable Q.The Q-Gram is extended beyond the length of the string with padding.
    
    Parameters:
    
    q - size of the tokens
    
    startPadding - padding to padd startof string with
    
    endPadding - padding to padd end of string with
    
    Returns:
    
    a q-gram tokenizer
  - transform
```
public static Tokenizer transform(Tokenizer tokenizer,
                                  com.google.common.base.Function<String,String> function)
```
    Constructs a new transforming tokenizer. After tokenization, all tokens are transformed by the function.
    
    Parameters:
    
    tokenizer - delegate tokenizer
    
    function - to transform tokens
    
    Returns:
    
    a new transforming tokenizer.
  - whitespace
```
public static Tokenizer whitespace()
```
    Returns a tokenizer that splits a string into tokens around whitespace. Does not return leading or trailing empty tokens.
    To create tokenizer that returns leading and trailing empty tokens use Tokenizers.pattern("\\s+")
    
    Returns:
    
    a white space tokenizer

Class Tokenizers

Method Summary

Methods inherited from class java.lang.Object

Method Detail

chain

chain

filter

pattern

pattern

qGram

qGramWithFilter

qGramWithPadding

qGramWithPadding

qGramWithPadding

transform

whitespace