Package org.apache.lucene.index

Class DocumentsWriterDeleteQueue

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, Accountable
    final class DocumentsWriterDeleteQueue
extends java.lang.Object
implements Accountable, java.io.Closeable
    DocumentsWriterDeleteQueue is a non-blocking linked pending deletes queue. In contrast to other queue implementation we only maintain the tail of the queue. A delete queue is always used in a context of a set of DWPTs and a global delete pool. Each of the DWPT and the global pool need to maintain their 'own' head of the queue (as a DeleteSlice instance per DocumentsWriterPerThread). The difference between the DWPT and the global pool is that the DWPT starts maintaining a head once it has added its first document since for its segments private deletes only the deletes after that document are relevant. The global pool instead starts maintaining the head once this instance is created by taking the sentinel instance as its initial head.

    Since each DocumentsWriterDeleteQueue.DeleteSlice maintains its own head and the list is only single linked the garbage collector takes care of pruning the list for us. All nodes in the list that are still relevant should be either directly or indirectly referenced by one of the DWPT's private DocumentsWriterDeleteQueue.DeleteSlice or by the global BufferedUpdates slice.

    Each DWPT as well as the global delete pool maintain their private DeleteSlice instance. In the DWPT case updating a slice is equivalent to atomically finishing the document. The slice update guarantees a "happens before" relationship to all other updates in the same indexing session. When a DWPT updates a document it:

    1. consumes a document and finishes its processing
    2. updates its private DocumentsWriterDeleteQueue.DeleteSlice either by calling updateSlice(DeleteSlice) or add(Node, DeleteSlice) (if the document has a delTerm)
    3. applies all deletes in the slice to its private BufferedUpdates and resets it
    4. increments its internal document id
    The DWPT also doesn't apply its current documents delete term until it has updated its delete slice which ensures the consistency of the update. If the update fails before the DeleteSlice could have been updated the deleteTerm will also not be added to its private deletes neither to the global deletes.

    • Field Detail

      • closed

        private volatile boolean closed

      • globalSlice

        private final DocumentsWriterDeleteQueue.DeleteSlice globalSlice
        Used to record deletes against all prior (already written to disk) segments. Whenever any segment flushes, we bundle up this set of deletes and insert into the buffered updates stream before the newly flushed segment(s).

      • globalBufferedUpdates

        private final BufferedUpdates globalBufferedUpdates

      • globalBufferLock

        final java.util.concurrent.locks.ReentrantLock globalBufferLock

      • generation

        final long generation

      • nextSeqNo

        private final java.util.concurrent.atomic.AtomicLong nextSeqNo
        Generates the sequence number that IW returns to callers changing the index, showing the effective serialization of all operations.

      • maxSeqNo

        private volatile long maxSeqNo

      • startSeqNo

        private final long startSeqNo

      • previousMaxSeqId

        private final java.util.function.LongSupplier previousMaxSeqId

      • advanced

        private boolean advanced

    • Constructor Detail

      • DocumentsWriterDeleteQueue

        DocumentsWriterDeleteQueue​(InfoStream infoStream)

      • DocumentsWriterDeleteQueue

        private DocumentsWriterDeleteQueue​(InfoStream infoStream,
                                   long generation,
                                   long startSeqNo,
                                   java.util.function.LongSupplier previousMaxSeqId)

    • Method Detail

      • addDelete

        long addDelete​(Query... queries)

      • addDelete

        long addDelete​(Term... terms)

      • addDocValuesUpdates

        long addDocValuesUpdates​(DocValuesUpdate... updates)

      • anyChanges

        boolean anyChanges()

      • tryApplyGlobalSlice

        void tryApplyGlobalSlice()

      • maybeFreezeGlobalBuffer

        FrozenBufferedUpdates maybeFreezeGlobalBuffer()
        This may freeze the global buffer unless the delete queue has already been closed. If the queue has been closed this method will return null

      • ensureOpen

        private void ensureOpen()

      • isOpen

        public boolean isOpen()

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable

      • numGlobalTermDeletes

        public int numGlobalTermDeletes()

      • clear

        void clear()

      • forceApplyGlobalSlice

        private boolean forceApplyGlobalSlice()

      • getBufferedUpdatesTermsSize

        public int getBufferedUpdatesTermsSize()

      • ramBytesUsed

        public long ramBytesUsed()
        Description copied from interface: Accountable
        Return the memory usage of this object in bytes. Negative values are illegal.
        Specified by:
        ramBytesUsed in interface Accountable

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object

      • getNextSequenceNumber

        public long getNextSequenceNumber()

      • getLastSequenceNumber

        long getLastSequenceNumber()

      • skipSequenceNumbers

        void skipSequenceNumbers​(long jump)
        Inserts a gap in the sequence numbers. This is used by IW during flush or commit to ensure any in-flight threads get sequence numbers inside the gap

      • getMaxCompletedSeqNo

        long getMaxCompletedSeqNo()
        Returns the maximum completed seq no for this queue.

      • getPrevMaxSeqIdSupplier

        private static java.util.function.LongSupplier getPrevMaxSeqIdSupplier​(java.util.concurrent.atomic.AtomicLong nextSeqNo)

      • advanceQueue

        DocumentsWriterDeleteQueue advanceQueue​(int maxNumPendingOps)
        Advances the queue to the next queue on flush. This carries over the the generation to the next queue and set the getMaxSeqNo() based on the given maxNumPendingOps. This method can only be called once, subsequently the returned queue should be used.
        Parameters:
        maxNumPendingOps - the max number of possible concurrent operations that will execute on this queue after it was advanced. This corresponds the the number of DWPTs that own the current queue at the moment when this queue is advanced since each these DWPTs can increment the seqId after we advanced it.
        Returns:
        a new queue as a successor of this queue.

      • getMaxSeqNo

        long getMaxSeqNo()
        Returns the maximum sequence number for this queue. This value will change once this queue is advanced.

      • isAdvanced

        boolean isAdvanced()
        Returns true if it was advanced.