Package org.apache.lucene.index

Class CheckIndex

java.lang.Object

org.apache.lucene.index.CheckIndex

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public final class CheckIndex
extends java.lang.Object
implements java.io.Closeable

Basic tool and API to check the health of an index and write a new segments file that removes reference to problematic segments.

As this tool checks every byte in the index, on a large index it can take quite a long time to run.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	CheckIndex.ConstantRelationIntersectVisitor
private static interface	CheckIndex.DocValuesIteratorSupplier
static class	CheckIndex.Options	Run-time configuration options for CheckIndex commands.
static class	CheckIndex.Status	Returned from checkIndex() detailing the health and status of the index.
static class	CheckIndex.VerifyPointsVisitor	Walks the entire N-dimensional points space, verifying that all points fall within the last cell's boundaries.

Field Summary

Fields

Modifier and Type	Field	Description
private static boolean	assertsOn
private boolean	checksumsOnly
private boolean	closed
private Directory	dir
private boolean	doSlowChecks
private boolean	failFast
private java.io.PrintStream	infoStream
private boolean	verbose
private Lock	writeLock

Constructor Summary

Constructors

Constructor	Description
CheckIndex(Directory dir)	Create a new CheckIndex on the directory.
CheckIndex(Directory dir, Lock writeLock)	Expert: create a directory with the specified lock.

Method Summary

Modifier and Type	Method	Description
static boolean	assertsOn()	Check whether asserts are enabled or not.
private static void	checkBinaryDocValues(java.lang.String fieldName, int maxDoc, BinaryDocValues bdv, BinaryDocValues bdv2)
private static void	checkDocValues(FieldInfo fi, DocValuesProducer dvReader, int maxDoc, java.io.PrintStream infoStream, CheckIndex.Status.DocValuesStatus status)
private static void	checkDVIterator(FieldInfo fi, int maxDoc, CheckIndex.DocValuesIteratorSupplier producer)
private static CheckIndex.Status.TermIndexStatus	checkFields(Fields fields, Bits liveDocs, int maxDoc, FieldInfos fieldInfos, NormsProducer normsProducer, boolean doPrint, boolean isVectors, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks)	checks Fields api is consistent with itself.
(package private) static void	checkImpacts(Impacts impacts, int lastTarget)
CheckIndex.Status	checkIndex()	Returns a CheckIndex.Status instance detailing the state of the index.
CheckIndex.Status	checkIndex(java.util.List<java.lang.String> onlySegments)	Returns a CheckIndex.Status instance detailing the state of the index.
private static void	checkNumericDocValues(java.lang.String fieldName, NumericDocValues ndv, NumericDocValues ndv2)
private static boolean	checkSingleTermRange(java.lang.String field, int maxDoc, Terms terms, BytesRef minTerm, BytesRef maxTerm, FixedBitSet normalDocs, FixedBitSet intersectDocs)	Test Terms.intersect on this range, and validates that it returns the same doc ids as using non-intersect TermsEnum.
private static void	checkSoftDeletes(java.lang.String softDeletesField, SegmentCommitInfo info, SegmentReader reader, java.io.PrintStream infoStream, boolean failFast)
private static void	checkSortedDocValues(java.lang.String fieldName, int maxDoc, SortedDocValues dv, SortedDocValues dv2)
private static void	checkSortedNumericDocValues(java.lang.String fieldName, int maxDoc, SortedNumericDocValues ndv, SortedNumericDocValues ndv2)
private static void	checkSortedSetDocValues(java.lang.String fieldName, int maxDoc, SortedSetDocValues dv, SortedSetDocValues dv2)
void	close()
int	doCheck(CheckIndex.Options opts)	Actually perform the index check
private static int	doMain(java.lang.String[] args)
boolean	doSlowChecks()	See setDoSlowChecks(boolean).
private void	ensureOpen()
void	exorciseIndex(CheckIndex.Status result)	Repairs the index using previously returned result from checkIndex().
boolean	getChecksumsOnly()	See getChecksumsOnly().
private static long	getDocsFromTermRange(java.lang.String field, int maxDoc, TermsEnum termsEnum, FixedBitSet docsSeen, BytesRef minTerm, BytesRef maxTerm, boolean isIntersect)	Visits all terms in the range minTerm (inclusive) to maxTerm (exclusive), marking all doc IDs encountered into allDocsSeen, and returning the total number of terms visited.
boolean	getFailFast()	See setFailFast(boolean).
static void	main(java.lang.String[] args)	Command-line interface to check and exorcise corrupt segments from an index.
private static void	msg(java.io.PrintStream out, java.lang.String msg)
private static double	nsToSec(long ns)
static CheckIndex.Options	parseOptions(java.lang.String[] args)	Parse command line args into fields
void	setChecksumsOnly(boolean v)	If true, only validate physical integrity for all files.
void	setDoSlowChecks(boolean v)	If true, additional slow checks are performed.
void	setFailFast(boolean v)	If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.
void	setInfoStream(java.io.PrintStream out)	Set infoStream where messages should go.
void	setInfoStream(java.io.PrintStream out, boolean verbose)	Set infoStream where messages should go.
private static boolean	testAsserts()
static CheckIndex.Status.DocValuesStatus	testDocValues(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)	Test docvalues.
static CheckIndex.Status.FieldInfoStatus	testFieldInfos(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)	Test field infos.
static CheckIndex.Status.FieldNormStatus	testFieldNorms(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)	Test field norms.
static CheckIndex.Status.LiveDocStatus	testLiveDocs(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)	Test live docs.
static CheckIndex.Status.PointsStatus	testPoints(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)	Test the points index
static CheckIndex.Status.TermIndexStatus	testPostings(CodecReader reader, java.io.PrintStream infoStream)	Test the term index.
static CheckIndex.Status.TermIndexStatus	testPostings(CodecReader reader, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)	Test the term index.
static CheckIndex.Status.IndexSortStatus	testSort(CodecReader reader, Sort sort, java.io.PrintStream infoStream, boolean failFast)	Tests index sort order.
static CheckIndex.Status.StoredFieldStatus	testStoredFields(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)	Test stored fields.
static CheckIndex.Status.TermVectorStatus	testTermVectors(CodecReader reader, java.io.PrintStream infoStream)	Test term vectors.
static CheckIndex.Status.TermVectorStatus	testTermVectors(CodecReader reader, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)	Test term vectors.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

infoStream

private java.io.PrintStream infoStream

dir

private Directory dir

writeLock

private Lock writeLock

closed

private volatile boolean closed

doSlowChecks

private boolean doSlowChecks

failFast

private boolean failFast

verbose

private boolean verbose

checksumsOnly

private boolean checksumsOnly

assertsOn

private static boolean assertsOn

Constructor Detail

CheckIndex

public CheckIndex(Directory dir)
           throws java.io.IOException

Create a new CheckIndex on the directory.

Throws:

java.io.IOException

CheckIndex

public CheckIndex(Directory dir,
                  Lock writeLock)

Expert: create a directory with the specified lock. This should really not be used except for unit tests!!!! It exists only to support special tests (such as TestIndexWriterExceptions*), that would otherwise be more complicated to debug if they had to close the writer for each check.

Method Detail

ensureOpen

private void ensureOpen()

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

setDoSlowChecks

public void setDoSlowChecks(boolean v)

If true, additional slow checks are performed. This will likely drastically increase time it takes to run CheckIndex!

doSlowChecks

public boolean doSlowChecks()

See setDoSlowChecks(boolean).

setFailFast

public void setFailFast(boolean v)

If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.

getFailFast

public boolean getFailFast()

See setFailFast(boolean).

getChecksumsOnly

public boolean getChecksumsOnly()

See getChecksumsOnly().

setChecksumsOnly

public void setChecksumsOnly(boolean v)

If true, only validate physical integrity for all files. Note that the returned nested status objects (e.g. storedFieldStatus) will be null.

setInfoStream

public void setInfoStream(java.io.PrintStream out,
                          boolean verbose)

Set infoStream where messages should go. If null, no messages are printed. If verbose is true then more details are printed.

setInfoStream

public void setInfoStream(java.io.PrintStream out)

Set infoStream where messages should go. See setInfoStream(PrintStream,boolean).

msg

private static void msg(java.io.PrintStream out,
                        java.lang.String msg)

checkIndex

public CheckIndex.Status checkIndex()
                             throws java.io.IOException

Returns a CheckIndex.Status instance detailing the state of the index.

As this method checks every byte in the index, on a large index it can take quite a long time to run.

WARNING: make sure you only call this when the index is not opened by any writer.

Throws:

java.io.IOException

checkIndex

public CheckIndex.Status checkIndex(java.util.List<java.lang.String> onlySegments)
                             throws java.io.IOException

Returns a CheckIndex.Status instance detailing the state of the index.

Parameters:

onlySegments - list of specific segment names to check

As this method checks every byte in the specified segments, on a large index it can take quite a long time to run.

Throws:

java.io.IOException

testSort

public static CheckIndex.Status.IndexSortStatus testSort(CodecReader reader,
                                                         Sort sort,
                                                         java.io.PrintStream infoStream,
                                                         boolean failFast)
                                                  throws java.io.IOException

Tests index sort order.

Throws:

java.io.IOException

testLiveDocs

public static CheckIndex.Status.LiveDocStatus testLiveDocs(CodecReader reader,
                                                           java.io.PrintStream infoStream,
                                                           boolean failFast)
                                                    throws java.io.IOException

Test live docs.

Throws:

java.io.IOException

testFieldInfos

public static CheckIndex.Status.FieldInfoStatus testFieldInfos(CodecReader reader,
                                                               java.io.PrintStream infoStream,
                                                               boolean failFast)
                                                        throws java.io.IOException

Test field infos.

Throws:

java.io.IOException

testFieldNorms

public static CheckIndex.Status.FieldNormStatus testFieldNorms(CodecReader reader,
                                                               java.io.PrintStream infoStream,
                                                               boolean failFast)
                                                        throws java.io.IOException

Test field norms.

Throws:

java.io.IOException

getDocsFromTermRange

private static long getDocsFromTermRange(java.lang.String field,
                                         int maxDoc,
                                         TermsEnum termsEnum,
                                         FixedBitSet docsSeen,
                                         BytesRef minTerm,
                                         BytesRef maxTerm,
                                         boolean isIntersect)
                                  throws java.io.IOException

Visits all terms in the range minTerm (inclusive) to maxTerm (exclusive), marking all doc IDs encountered into allDocsSeen, and returning the total number of terms visited.

Throws:

java.io.IOException

checkSingleTermRange

private static boolean checkSingleTermRange(java.lang.String field,
                                            int maxDoc,
                                            Terms terms,
                                            BytesRef minTerm,
                                            BytesRef maxTerm,
                                            FixedBitSet normalDocs,
                                            FixedBitSet intersectDocs)
                                     throws java.io.IOException

Test Terms.intersect on this range, and validates that it returns the same doc ids as using non-intersect TermsEnum. Returns true if any fake terms were seen.

Throws:

java.io.IOException

checkFields

private static CheckIndex.Status.TermIndexStatus checkFields(Fields fields,
                                                             Bits liveDocs,
                                                             int maxDoc,
                                                             FieldInfos fieldInfos,
                                                             NormsProducer normsProducer,
                                                             boolean doPrint,
                                                             boolean isVectors,
                                                             java.io.PrintStream infoStream,
                                                             boolean verbose,
                                                             boolean doSlowChecks)
                                                      throws java.io.IOException

checks Fields api is consistent with itself. searcher is optional, to verify with queries. Can be null.

Throws:

java.io.IOException

checkImpacts

static void checkImpacts(Impacts impacts,
                         int lastTarget)

testPostings

public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader,
                                                             java.io.PrintStream infoStream)
                                                      throws java.io.IOException

Test the term index.

Throws:

java.io.IOException

testPostings

public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader,
                                                             java.io.PrintStream infoStream,
                                                             boolean verbose,
                                                             boolean doSlowChecks,
                                                             boolean failFast)
                                                      throws java.io.IOException

Test the term index.

Throws:

java.io.IOException

testPoints

public static CheckIndex.Status.PointsStatus testPoints(CodecReader reader,
                                                        java.io.PrintStream infoStream,
                                                        boolean failFast)
                                                 throws java.io.IOException

Test the points index

Throws:

java.io.IOException

testStoredFields

public static CheckIndex.Status.StoredFieldStatus testStoredFields(CodecReader reader,
                                                                   java.io.PrintStream infoStream,
                                                                   boolean failFast)
                                                            throws java.io.IOException

Test stored fields.

Throws:

java.io.IOException

testDocValues

public static CheckIndex.Status.DocValuesStatus testDocValues(CodecReader reader,
                                                              java.io.PrintStream infoStream,
                                                              boolean failFast)
                                                       throws java.io.IOException

Test docvalues.

Throws:

java.io.IOException

checkDVIterator

private static void checkDVIterator(FieldInfo fi,
                                    int maxDoc,
                                    CheckIndex.DocValuesIteratorSupplier producer)
                             throws java.io.IOException

Throws:

java.io.IOException

checkBinaryDocValues

private static void checkBinaryDocValues(java.lang.String fieldName,
                                         int maxDoc,
                                         BinaryDocValues bdv,
                                         BinaryDocValues bdv2)
                                  throws java.io.IOException

Throws:

java.io.IOException

checkSortedDocValues

private static void checkSortedDocValues(java.lang.String fieldName,
                                         int maxDoc,
                                         SortedDocValues dv,
                                         SortedDocValues dv2)
                                  throws java.io.IOException

Throws:

java.io.IOException

checkSortedSetDocValues

private static void checkSortedSetDocValues(java.lang.String fieldName,
                                            int maxDoc,
                                            SortedSetDocValues dv,
                                            SortedSetDocValues dv2)
                                     throws java.io.IOException

Throws:

java.io.IOException

checkSortedNumericDocValues

private static void checkSortedNumericDocValues(java.lang.String fieldName,
                                                int maxDoc,
                                                SortedNumericDocValues ndv,
                                                SortedNumericDocValues ndv2)
                                         throws java.io.IOException

Throws:

java.io.IOException

checkNumericDocValues

private static void checkNumericDocValues(java.lang.String fieldName,
                                          NumericDocValues ndv,
                                          NumericDocValues ndv2)
                                   throws java.io.IOException

Throws:

java.io.IOException

checkDocValues

private static void checkDocValues(FieldInfo fi,
                                   DocValuesProducer dvReader,
                                   int maxDoc,
                                   java.io.PrintStream infoStream,
                                   CheckIndex.Status.DocValuesStatus status)
                            throws java.lang.Exception

Throws:

java.lang.Exception

testTermVectors

public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader,
                                                                 java.io.PrintStream infoStream)
                                                          throws java.io.IOException

Test term vectors.

Throws:

java.io.IOException

testTermVectors

public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader,
                                                                 java.io.PrintStream infoStream,
                                                                 boolean verbose,
                                                                 boolean doSlowChecks,
                                                                 boolean failFast)
                                                          throws java.io.IOException

Test term vectors.

Throws:

java.io.IOException

exorciseIndex

public void exorciseIndex(CheckIndex.Status result)
                   throws java.io.IOException

Repairs the index using previously returned result from checkIndex(). Note that this does not remove any of the unreferenced files after it's done; you must separately open an IndexWriter, which deletes unreferenced files when it's created.

WARNING: this writes a new segments file into the index, effectively removing all documents in broken segments from the index. BE CAREFUL.

Throws:

java.io.IOException

testAsserts

private static boolean testAsserts()

assertsOn

public static boolean assertsOn()

Check whether asserts are enabled or not.

Returns:

true iff asserts are enabled

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException,
                        java.lang.InterruptedException

Command-line interface to check and exorcise corrupt segments from an index.

Run it like this:

    java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-exorcise] [-verbose] [-segment X] [-segment Y]
    

• -exorcise: actually write a new segments_N file, removing any problematic segments. *LOSES DATA*
• -segment X: only check the specified segment(s). This can be specified multiple times, to check more than one segment, eg -segment _2 -segment _a. You can't use this with the -exorcise option.

WARNING: -exorcise should only be used on an emergency basis as it will cause documents (perhaps many) to be permanently removed from the index. Always make a backup copy of your index before running this! Do not run this tool on an index that is actively being written to. You have been warned!

Run without -exorcise, this tool will open the index, report version information and report any exceptions it hits and what action it would take if -exorcise were specified. With -exorcise, this tool will remove any segments that have issues and write a new segments_N file. This means all documents contained in the affected segments will be removed.

This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0.

Throws:

java.io.IOException

java.lang.InterruptedException

doMain

private static int doMain(java.lang.String[] args)
                   throws java.io.IOException,
                          java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException

parseOptions

public static CheckIndex.Options parseOptions(java.lang.String[] args)

Parse command line args into fields

Parameters:

args - The command line arguments

Returns:

An Options struct

Throws:

java.lang.IllegalArgumentException - if any of the CLI args are invalid

doCheck

public int doCheck(CheckIndex.Options opts)
            throws java.io.IOException,
                   java.lang.InterruptedException

Actually perform the index check

Parameters:

opts - The options to use for this check

Returns:

0 iff the index is clean, 1 otherwise

Throws:

java.io.IOException

java.lang.InterruptedException

checkSoftDeletes

private static void checkSoftDeletes(java.lang.String softDeletesField,
                                     SegmentCommitInfo info,
                                     SegmentReader reader,
                                     java.io.PrintStream infoStream,
                                     boolean failFast)
                              throws java.io.IOException

Throws:

java.io.IOException

nsToSec

private static double nsToSec(long ns)