Package org.apache.lucene.analysis.ko

Class KoreanTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.ko.KoreanTokenizer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public final class KoreanTokenizer
extends Tokenizer

Tokenizer for Korean that uses morphological analysis.

This tokenizer sets a number of additional attributes:

• PartOfSpeechAttribute containing part-of-speech.
• ReadingAttribute containing reading.

This tokenizer uses a rolling Viterbi search to find the least cost segmentation (path) of the incoming characters.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KoreanTokenizer.DecompoundMode	Decompound mode: this determines how the tokenizer handles POS.Type.COMPOUND, POS.Type.INFLECT and POS.Type.PREANALYSIS tokens.
(package private) static class	KoreanTokenizer.Position
static class	KoreanTokenizer.Type	Token type reflecting the original source of this token
(package private) static class	KoreanTokenizer.WrappedPositionArray

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field	Description
private FST.Arc<java.lang.Long>	arc
private RollingCharBuffer	buffer
private CharacterDefinition	characterDefinition
private ConnectionCosts	costs
static KoreanTokenizer.DecompoundMode	DEFAULT_DECOMPOUND	Default mode for the decompound of tokens (KoreanTokenizer.DecompoundMode.DISCARD.
private TokenInfoDictionary	dictionary
private java.util.EnumMap<KoreanTokenizer.Type,Dictionary>	dictionaryMap
private boolean	discardPunctuation
private GraphvizFormatter	dotOut
private boolean	end
private TokenInfoFST	fst
private FST.BytesReader	fstReader
private int	lastBackTracePos
private static int	MAX_BACKTRACE_GAP
private static int	MAX_UNKNOWN_WORD_LENGTH
private KoreanTokenizer.DecompoundMode	mode
private OffsetAttribute	offsetAtt
private boolean	outputUnknownUnigrams
private java.util.List<Token>	pending
private int	pos
private PartOfSpeechAttribute	posAtt
private PositionIncrementAttribute	posIncAtt
private KoreanTokenizer.WrappedPositionArray	positions
private PositionLengthAttribute	posLengthAtt
private ReadingAttribute	readingAtt
private CharTermAttribute	termAtt
private UnknownDictionary	unkDictionary
private UserDictionary	userDictionary
private TokenInfoFST	userFST
private FST.BytesReader	userFSTReader
private static boolean	VERBOSE
private IntsRef	wordIdRef

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
KoreanTokenizer()	Creates a new KoreanTokenizer with default parameters.
KoreanTokenizer(AttributeFactory factory, TokenInfoDictionary systemDictionary, UnknownDictionary unkDictionary, ConnectionCosts connectionCosts, UserDictionary userDictionary, KoreanTokenizer.DecompoundMode mode, boolean outputUnknownUnigrams, boolean discardPunctuation)	Create a new KoreanTokenizer supplying a custom system dictionary and unknown dictionary.
KoreanTokenizer(AttributeFactory factory, UserDictionary userDictionary, KoreanTokenizer.DecompoundMode mode, boolean outputUnknownUnigrams)	Create a new KoreanTokenizer using the system and unknown dictionaries shipped with Lucene.
KoreanTokenizer(AttributeFactory factory, UserDictionary userDictionary, KoreanTokenizer.DecompoundMode mode, boolean outputUnknownUnigrams, boolean discardPunctuation)	Create a new KoreanTokenizer using the system and unknown dictionaries shipped with Lucene.

Method Summary

Modifier and Type	Method	Description
private void	add(Dictionary dict, KoreanTokenizer.Position fromPosData, int wordPos, int endPos, int wordID, KoreanTokenizer.Type type)
private void	backtrace(KoreanTokenizer.Position endPosData, int fromIDX)
void	close()	Releases resources associated with this stream.
private int	computeSpacePenalty(POS.Tag leftPOS, int numSpaces)	Returns the space penalty associated with the provided POS.Tag.
void	end()	This method is called by the consumer after the last token has been consumed, after TokenStream.incrementToken() returned false (using the new TokenStream API).
(package private) Dictionary	getDict(KoreanTokenizer.Type type)
boolean	incrementToken()	Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.
private static boolean	isCommonOrInherited(java.lang.Character.UnicodeScript script)
private static boolean	isPunctuation(char ch)
private static boolean	isPunctuation(char ch, int cid)
private static boolean	isSameScript(java.lang.Character.UnicodeScript scriptOne, java.lang.Character.UnicodeScript scriptTwo)	Determine if two scripts are compatible.
private void	parse()
void	reset()	This method is called by a consumer before it begins consumption using TokenStream.incrementToken().
private void	resetState()
void	setGraphvizFormatter(GraphvizFormatter dotOut)	Expert: set this to produce graphviz (dot) output of the Viterbi lattice
private boolean	shouldFilterToken(Token token)

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_DECOMPOUND

public static final KoreanTokenizer.DecompoundMode DEFAULT_DECOMPOUND

Default mode for the decompound of tokens (KoreanTokenizer.DecompoundMode.DISCARD.

VERBOSE

private static final boolean VERBOSE

See Also:

Constant Field Values

MAX_UNKNOWN_WORD_LENGTH

private static final int MAX_UNKNOWN_WORD_LENGTH

See Also:

Constant Field Values

MAX_BACKTRACE_GAP

private static final int MAX_BACKTRACE_GAP

See Also:

Constant Field Values

dictionaryMap

private final java.util.EnumMap<KoreanTokenizer.Type,Dictionary> dictionaryMap

fst

private final TokenInfoFST fst

dictionary

private final TokenInfoDictionary dictionary

unkDictionary

private final UnknownDictionary unkDictionary

costs

private final ConnectionCosts costs

userDictionary

private final UserDictionary userDictionary

characterDefinition

private final CharacterDefinition characterDefinition

arc

private final FST.Arc<java.lang.Long> arc

fstReader

private final FST.BytesReader fstReader

wordIdRef

private final IntsRef wordIdRef

userFSTReader

private final FST.BytesReader userFSTReader

userFST

private final TokenInfoFST userFST

discardPunctuation

private final boolean discardPunctuation

mode

private final KoreanTokenizer.DecompoundMode mode

outputUnknownUnigrams

private final boolean outputUnknownUnigrams

buffer

private final RollingCharBuffer buffer

positions

private final KoreanTokenizer.WrappedPositionArray positions

end

private boolean end

lastBackTracePos

private int lastBackTracePos

pos

private int pos

pending

private final java.util.List<Token> pending

termAtt

private final CharTermAttribute termAtt

offsetAtt

private final OffsetAttribute offsetAtt

posIncAtt

private final PositionIncrementAttribute posIncAtt

posLengthAtt

private final PositionLengthAttribute posLengthAtt

posAtt

private final PartOfSpeechAttribute posAtt

readingAtt

private final ReadingAttribute readingAtt

dotOut

private GraphvizFormatter dotOut

Constructor Detail

KoreanTokenizer

public KoreanTokenizer()

Creates a new KoreanTokenizer with default parameters.

Uses the default AttributeFactory.

KoreanTokenizer

public KoreanTokenizer(AttributeFactory factory,
                       UserDictionary userDictionary,
                       KoreanTokenizer.DecompoundMode mode,
                       boolean outputUnknownUnigrams)

Create a new KoreanTokenizer using the system and unknown dictionaries shipped with Lucene.

Parameters:

factory - the AttributeFactory to use

userDictionary - Optional: if non-null, user dictionary.

mode - Decompound mode.

outputUnknownUnigrams - if true outputs unigrams for unknown words.

KoreanTokenizer

public KoreanTokenizer(AttributeFactory factory,
                       UserDictionary userDictionary,
                       KoreanTokenizer.DecompoundMode mode,
                       boolean outputUnknownUnigrams,
                       boolean discardPunctuation)

Create a new KoreanTokenizer using the system and unknown dictionaries shipped with Lucene.

Parameters:

factory - the AttributeFactory to use

userDictionary - Optional: if non-null, user dictionary.

mode - Decompound mode.

outputUnknownUnigrams - if true outputs unigrams for unknown words.

discardPunctuation - true if punctuation tokens should be dropped from the output.

KoreanTokenizer

public KoreanTokenizer(AttributeFactory factory,
                       TokenInfoDictionary systemDictionary,
                       UnknownDictionary unkDictionary,
                       ConnectionCosts connectionCosts,
                       UserDictionary userDictionary,
                       KoreanTokenizer.DecompoundMode mode,
                       boolean outputUnknownUnigrams,
                       boolean discardPunctuation)

Create a new KoreanTokenizer supplying a custom system dictionary and unknown dictionary. This constructor provides an entry point for users that want to construct custom language models that can be used as input to DictionaryBuilder.

Parameters:

factory - the AttributeFactory to use

systemDictionary - a custom known token dictionary

unkDictionary - a custom unknown token dictionary

connectionCosts - custom token transition costs

userDictionary - Optional: if non-null, user dictionary.

mode - Decompound mode.

outputUnknownUnigrams - if true outputs unigrams for unknown words.

discardPunctuation - true if punctuation tokens should be dropped from the output.

Method Detail

setGraphvizFormatter

public void setGraphvizFormatter(GraphvizFormatter dotOut)

Expert: set this to produce graphviz (dot) output of the Viterbi lattice

close

public void close()
           throws java.io.IOException

Description copied from class: Tokenizer

Releases resources associated with this stream.

If you override this method, always call super.close(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on reuse).

NOTE: The default implementation closes the input Reader, so be sure to call super.close() when overriding this method.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Overrides:

close in class Tokenizer

Throws:

java.io.IOException

reset

public void reset()
           throws java.io.IOException

Description copied from class: TokenStream

This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).

Overrides:

reset in class Tokenizer

Throws:

java.io.IOException

resetState

private void resetState()

end

public void end()
         throws java.io.IOException

Description copied from class: TokenStream

This method is called by the consumer after the last token has been consumed, after TokenStream.incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.

If you override this method, always call super.end().

Overrides:

end in class TokenStream

Throws:

java.io.IOException - If an I/O error occurs

computeSpacePenalty

private int computeSpacePenalty(POS.Tag leftPOS,
                                int numSpaces)

Returns the space penalty associated with the provided POS.Tag.

Parameters:

leftPOS - the left part of speech of the current token.

numSpaces - the number of spaces before the current token.

add

private void add(Dictionary dict,
                 KoreanTokenizer.Position fromPosData,
                 int wordPos,
                 int endPos,
                 int wordID,
                 KoreanTokenizer.Type type)

incrementToken

public boolean incrementToken()
                       throws java.io.IOException

Description copied from class: TokenStream

Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in TokenStream.incrementToken().

Specified by:

incrementToken in class TokenStream

Returns:

false for end of stream; true otherwise

Throws:

java.io.IOException

parse

private void parse()
            throws java.io.IOException

Throws:

java.io.IOException

backtrace

private void backtrace(KoreanTokenizer.Position endPosData,
                       int fromIDX)

getDict

Dictionary getDict(KoreanTokenizer.Type type)

shouldFilterToken

private boolean shouldFilterToken(Token token)

isPunctuation

private static boolean isPunctuation(char ch)

isPunctuation

private static boolean isPunctuation(char ch,
                                     int cid)

isCommonOrInherited

private static boolean isCommonOrInherited(java.lang.Character.UnicodeScript script)

isSameScript

private static boolean isSameScript(java.lang.Character.UnicodeScript scriptOne,
                                    java.lang.Character.UnicodeScript scriptTwo)

Determine if two scripts are compatible.