Package org.apache.lucene.analysis.cn.smart

Class SmartChineseAnalyzer

java.lang.Object

org.apache.lucene.analysis.Analyzer

org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public final class SmartChineseAnalyzer
extends Analyzer

SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

Since:

3.1

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	SmartChineseAnalyzer.DefaultSetHolder	Atomically loads the DEFAULT_STOP_SET in a lazy fashion once the outer class accesses the static final set the first time.;

Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer

Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents

Field Summary

Fields

Modifier and Type	Field	Description
private static java.lang.String	DEFAULT_STOPWORD_FILE
private static java.lang.String	STOPWORD_FILE_COMMENT
private CharArraySet	stopWords

Fields inherited from class org.apache.lucene.analysis.Analyzer

GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY

Constructor Summary

Constructors

Constructor	Description
SmartChineseAnalyzer()	Create a new SmartChineseAnalyzer, using the default stopword list.
SmartChineseAnalyzer(boolean useDefaultStopWords)	Create a new SmartChineseAnalyzer, optionally using the default stopword list.
SmartChineseAnalyzer(CharArraySet stopWords)	Create a new SmartChineseAnalyzer, using the provided Set of stopwords.

Method Summary

Modifier and Type	Method	Description
Analyzer.TokenStreamComponents	createComponents(java.lang.String fieldName)	Creates a new Analyzer.TokenStreamComponents instance for this analyzer.
static CharArraySet	getDefaultStopSet()	Returns an unmodifiable instance of the default stop-words set.
protected TokenStream	normalize(java.lang.String fieldName, TokenStream in)	Wrap the given TokenStream in order to apply normalization filters.

Methods inherited from class org.apache.lucene.analysis.Analyzer

attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, getVersion, initReader, initReaderForNormalization, normalize, setVersion, tokenStream, tokenStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

stopWords

private final CharArraySet stopWords

DEFAULT_STOPWORD_FILE

private static final java.lang.String DEFAULT_STOPWORD_FILE

See Also:

Constant Field Values

STOPWORD_FILE_COMMENT

private static final java.lang.String STOPWORD_FILE_COMMENT

See Also:

Constant Field Values

Constructor Detail

SmartChineseAnalyzer

public SmartChineseAnalyzer()

Create a new SmartChineseAnalyzer, using the default stopword list.

SmartChineseAnalyzer

public SmartChineseAnalyzer(boolean useDefaultStopWords)

Create a new SmartChineseAnalyzer, optionally using the default stopword list.

The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

Parameters:

useDefaultStopWords - true to use the default stopword list.

SmartChineseAnalyzer

public SmartChineseAnalyzer(CharArraySet stopWords)

Create a new SmartChineseAnalyzer, using the provided Set of stopwords.

Note: the set should include punctuation, unless you want to index punctuation!

Parameters:

stopWords - Set of stopwords to use.

Method Detail

getDefaultStopSet

public static CharArraySet getDefaultStopSet()

Returns an unmodifiable instance of the default stop-words set.

Returns:

an unmodifiable instance of the default stop-words set.

createComponents

public Analyzer.TokenStreamComponents createComponents(java.lang.String fieldName)

Description copied from class: Analyzer

Creates a new Analyzer.TokenStreamComponents instance for this analyzer.

Specified by:

createComponents in class Analyzer

Parameters:

fieldName - the name of the fields content passed to the Analyzer.TokenStreamComponents sink as a reader

Returns:

the Analyzer.TokenStreamComponents for this analyzer.

normalize

protected TokenStream normalize(java.lang.String fieldName,
                                TokenStream in)

Description copied from class: Analyzer

Wrap the given TokenStream in order to apply normalization filters. The default implementation returns the TokenStream as-is. This is used by Analyzer.normalize(String, String).

Overrides:

normalize in class Analyzer