Package org.apache.lucene.util

Class OfflineSorter

java.lang.Object

org.apache.lucene.util.OfflineSorter

public class OfflineSorter
extends java.lang.Object

On-disk sorting of byte arrays. Each byte array (entry) is a composed of the following fields:

• (two bytes) length of the following byte array,
• exactly the above count of bytes for the sequence to be sorted.

See Also:

sort(String)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	OfflineSorter.BufferSize	A bit more descriptive unit for constructors.
static class	OfflineSorter.ByteSequencesReader	Utility class to read length-prefixed byte[] entries from an input.
static class	OfflineSorter.ByteSequencesWriter	Utility class to emit length-prefixed byte[] entries to an output stream for sorting.
(package private) static class	OfflineSorter.FileAndTop
private class	OfflineSorter.MergePartitionsTask	Merges multiple file-based partitions to a single on-disk partition.
private static class	OfflineSorter.Partition	Holds one partition of items, either loaded into memory or based on a file.
class	OfflineSorter.SortInfo	Sort info (debugging mostly).
private class	OfflineSorter.SortPartitionTask	Sorts one in-memory partition, writes it to disk, and returns the resulting file-based partition.

Field Summary

Fields

Modifier and Type	Field	Description
static long	ABSOLUTE_MIN_SORT_BUFFER_SIZE	Absolute minimum required buffer size for sorting.
private java.util.Comparator<BytesRef>	comparator
static java.util.Comparator<BytesRef>	DEFAULT_COMPARATOR	Default comparator: sorts in binary (codepoint) order
private Directory	dir
private java.util.concurrent.ExecutorService	exec
static long	GB	Convenience constant for gigabytes
static int	MAX_TEMPFILES	Maximum number of temporary files before doing an intermediate merge.
private int	maxTempFiles
static long	MB	Convenience constant for megabytes
static long	MIN_BUFFER_SIZE_MB	Minimum recommended buffer size for sorting.
private static java.lang.String	MIN_BUFFER_SIZE_MSG
private java.util.concurrent.Semaphore	partitionsInRAM
private OfflineSorter.BufferSize	ramBufferSize
(package private) OfflineSorter.SortInfo	sortInfo
private java.lang.String	tempFileNamePrefix
private int	valueLength

Constructor Summary

Constructors

Constructor	Description
OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix)	Defaults constructor.
OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix, java.util.Comparator<BytesRef> comparator)	Defaults constructor with a custom comparator.
OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix, java.util.Comparator<BytesRef> comparator, OfflineSorter.BufferSize ramBufferSize, int maxTempfiles, int valueLength, java.util.concurrent.ExecutorService exec, int maxPartitionsInRAM)	All-details constructor.

Method Summary

Modifier and Type	Method	Description
java.util.Comparator<BytesRef>	getComparator()	Returns the comparator in use to sort entries
Directory	getDirectory()	Returns the Directory we use to create temp files.
private OfflineSorter.Partition	getPartition(java.util.concurrent.Future<OfflineSorter.Partition> future)
protected OfflineSorter.ByteSequencesReader	getReader(ChecksumIndexInput in, java.lang.String name)	Subclasses can override to change how byte sequences are read from disk.
java.lang.String	getTempFileNamePrefix()	Returns the temp file name prefix passed to Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext) to generate temporary files.
protected OfflineSorter.ByteSequencesWriter	getWriter(IndexOutput out, long itemCount)	Subclasses can override to change how byte sequences are written to disk.
(package private) void	mergePartitions(Directory trackingDir, java.util.List<java.util.concurrent.Future<OfflineSorter.Partition>> segments)	Merge the most recent maxTempFile partitions into a new partition.
(package private) OfflineSorter.Partition	readPartition(OfflineSorter.ByteSequencesReader reader)	Read in a single partition of data, setting isExhausted[0] to true if there are no more items.
java.lang.String	sort(java.lang.String inputFileName)	Sort input to a new temp file, returning its name.
private void	verifyChecksum(java.lang.Throwable priorException, OfflineSorter.ByteSequencesReader reader)	Called on exception, to check whether the checksum is also corrupt in this source, and add that information (checksum matched or didn't) as a suppressed exception.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MB

public static final long MB

Convenience constant for megabytes

See Also:

Constant Field Values

GB

public static final long GB

Convenience constant for gigabytes

See Also:

Constant Field Values

MIN_BUFFER_SIZE_MB

public static final long MIN_BUFFER_SIZE_MB

Minimum recommended buffer size for sorting.

See Also:

Constant Field Values

ABSOLUTE_MIN_SORT_BUFFER_SIZE

public static final long ABSOLUTE_MIN_SORT_BUFFER_SIZE

Absolute minimum required buffer size for sorting.

See Also:

Constant Field Values

MIN_BUFFER_SIZE_MSG

private static final java.lang.String MIN_BUFFER_SIZE_MSG

See Also:

Constant Field Values

MAX_TEMPFILES

public static final int MAX_TEMPFILES

Maximum number of temporary files before doing an intermediate merge.

See Also:

Constant Field Values

dir

private final Directory dir

valueLength

private final int valueLength

tempFileNamePrefix

private final java.lang.String tempFileNamePrefix

exec

private final java.util.concurrent.ExecutorService exec

partitionsInRAM

private final java.util.concurrent.Semaphore partitionsInRAM

ramBufferSize

private final OfflineSorter.BufferSize ramBufferSize

sortInfo

OfflineSorter.SortInfo sortInfo

maxTempFiles

private int maxTempFiles

comparator

private final java.util.Comparator<BytesRef> comparator

DEFAULT_COMPARATOR

public static final java.util.Comparator<BytesRef> DEFAULT_COMPARATOR

Default comparator: sorts in binary (codepoint) order

Constructor Detail

OfflineSorter

public OfflineSorter(Directory dir,
                     java.lang.String tempFileNamePrefix)
              throws java.io.IOException

Defaults constructor.

Throws:

java.io.IOException

See Also:

OfflineSorter.BufferSize.automatic()

OfflineSorter

public OfflineSorter(Directory dir,
                     java.lang.String tempFileNamePrefix,
                     java.util.Comparator<BytesRef> comparator)
              throws java.io.IOException

Defaults constructor with a custom comparator.

Throws:

java.io.IOException

See Also:

OfflineSorter.BufferSize.automatic()

OfflineSorter

public OfflineSorter(Directory dir,
                     java.lang.String tempFileNamePrefix,
                     java.util.Comparator<BytesRef> comparator,
                     OfflineSorter.BufferSize ramBufferSize,
                     int maxTempfiles,
                     int valueLength,
                     java.util.concurrent.ExecutorService exec,
                     int maxPartitionsInRAM)

All-details constructor. If valueLength is -1 (the default), the length of each value differs; otherwise, all values have the specified length. If you pass a non-null ExecutorService then it will be used to run sorting operations that can be run concurrently, and maxPartitionsInRAM is the maximum concurrent in-memory partitions. Thus the maximum possible RAM used by this class while sorting is maxPartitionsInRAM * ramBufferSize.

Method Detail

getDirectory

public Directory getDirectory()

Returns the Directory we use to create temp files.

getTempFileNamePrefix

public java.lang.String getTempFileNamePrefix()

Returns the temp file name prefix passed to Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext) to generate temporary files.

sort

public java.lang.String sort(java.lang.String inputFileName)
                      throws java.io.IOException

Sort input to a new temp file, returning its name.

Throws:

java.io.IOException

verifyChecksum

private void verifyChecksum(java.lang.Throwable priorException,
                            OfflineSorter.ByteSequencesReader reader)
                     throws java.io.IOException

Called on exception, to check whether the checksum is also corrupt in this source, and add that information (checksum matched or didn't) as a suppressed exception.

Throws:

java.io.IOException

mergePartitions

void mergePartitions(Directory trackingDir,
                     java.util.List<java.util.concurrent.Future<OfflineSorter.Partition>> segments)
              throws java.io.IOException

Merge the most recent maxTempFile partitions into a new partition.

Throws:

java.io.IOException

readPartition

OfflineSorter.Partition readPartition(OfflineSorter.ByteSequencesReader reader)
                               throws java.io.IOException,
                                      java.lang.InterruptedException

Read in a single partition of data, setting isExhausted[0] to true if there are no more items.

Throws:

java.io.IOException

java.lang.InterruptedException

getWriter

protected OfflineSorter.ByteSequencesWriter getWriter(IndexOutput out,
                                                      long itemCount)
                                               throws java.io.IOException

Subclasses can override to change how byte sequences are written to disk.

Throws:

java.io.IOException

getReader

protected OfflineSorter.ByteSequencesReader getReader(ChecksumIndexInput in,
                                                      java.lang.String name)
                                               throws java.io.IOException

Subclasses can override to change how byte sequences are read from disk.

Throws:

java.io.IOException

getComparator

public java.util.Comparator<BytesRef> getComparator()

Returns the comparator in use to sort entries

getPartition

private OfflineSorter.Partition getPartition(java.util.concurrent.Future<OfflineSorter.Partition> future)
                                      throws java.io.IOException

Throws:

java.io.IOException