Package org.apache.lucene.analysis.icu

Class ICUTransformFilter

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    public final class ICUTransformFilter
extends TokenFilter
    A TokenFilter that transforms text with ICU.

    ICU provides text-transformation functionality via its Transliteration API. Although script conversion is its most common use, a Transliterator can actually perform a more general class of tasks. In fact, Transliterator defines a very general API which specifies only that a segment of the input text is replaced by new text. The particulars of this conversion are determined entirely by subclasses of Transliterator.

    Some useful transformations for search are built-in:

    • Conversion from Traditional to Simplified Chinese characters
    • Conversion from Hiragana to Katakana
    • Conversion from Fullwidth to Halfwidth forms.
    • Script conversions, for example Serbian Cyrillic to Latin

    Example usage:

    stream = new ICUTransformFilter(stream, Transliterator.getInstance("Traditional-Simplified"));

    For more details, see the ICU User Guide.

    • Constructor Detail

      • ICUTransformFilter

        public ICUTransformFilter​(TokenStream input,
                          com.ibm.icu.text.Transliterator transform)
        Create a new ICUTransformFilter that transforms text on the given stream.
        Parameters:
        input - TokenStream to filter.
        transform - Transliterator to transform the text.

    • Method Detail

      • incrementToken

        public boolean incrementToken()
                       throws java.io.IOException
        Description copied from class: TokenStream
        Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

        The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.

        This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.

        To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in TokenStream.incrementToken().

        Specified by:
        incrementToken in class TokenStream
        Returns:
        false for end of stream; true otherwise
        Throws:
        java.io.IOException