Package org.apache.lucene.store

Class FSDirectory

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    MMapDirectory, NativeUnixDirectory, NIOFSDirectory, RAFDirectory, SimpleFSDirectory, WindowsDirectory
    public abstract class FSDirectory
extends BaseDirectory
    Base class for Directory implementations that store index files in the file system. There are currently three core subclasses:
    • SimpleFSDirectory is a straightforward implementation using Files.newByteChannel. However, it has poor concurrent performance (multiple threads will bottleneck) as it synchronizes when multiple threads read from the same file.
    • NIOFSDirectory uses java.nio's FileChannel's positional io when reading to avoid synchronization when reading from the same file. Unfortunately, due to a Windows-only Sun JRE bug this is a poor choice for Windows, but on all other platforms this is the preferred choice. Applications using Thread.interrupt() or Future.cancel(boolean) should use RAFDirectory instead. See NIOFSDirectory java doc for details.
    • MMapDirectory uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space. Java has currently the limitation of not being able to unmap files from user code. The files are unmapped, when GC releases the byte buffers. Due to this bug in Sun's JRE, MMapDirectory's IndexInput.close() is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed. This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of. This class supplies a (possibly dangerous) workaround mentioned in the bug report, which may fail on non-Sun JVMs.

    Unfortunately, because of system peculiarities, there is no single overall best implementation. Therefore, we've added the open(java.nio.file.Path) method, to allow Lucene to choose the best FSDirectory implementation given your environment, and the known limitations of each implementation. For users who have no reason to prefer a specific implementation, it's best to simply use open(java.nio.file.Path). For all others, you should instantiate the desired implementation directly.

    NOTE: Accessing one of the above subclasses either directly or indirectly from a thread while it's interrupted can close the underlying channel immediately if at the same time the thread is blocked on IO. The channel will remain closed and subsequent access to the index will throw a ClosedChannelException. Applications using Thread.interrupt() or Future.cancel(boolean) should use the slower legacy RAFDirectory from the misc Lucene module instead.

    The locking implementation is by default NativeFSLockFactory, but can be changed by passing in a custom LockFactory instance.

    See Also:
    Directory

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      (package private) class  FSDirectory.FSIndexOutput  

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected FSDirectory​(java.nio.file.Path path, LockFactory lockFactory)
      Create a new FSDirectory for the named location (ctor for subclasses).

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()
      Closes the directory.
      IndexOutput createOutput​(java.lang.String name, IOContext context)
      Creates a new, empty file in the directory and returns an IndexOutput instance for appending data to this file.
      IndexOutput createTempOutput​(java.lang.String prefix, java.lang.String suffix, IOContext context)
      Creates a new, empty, temporary file in the directory and returns an IndexOutput instance for appending data to this file.
      void deleteFile​(java.lang.String name)
      Removes an existing file in the directory.
      void deletePendingFiles()
      Try to delete any pending files that we had previously tried to delete but failed because we are on Windows and the files were still held open.
      protected void ensureCanRead​(java.lang.String name)  
      long fileLength​(java.lang.String name)
      Returns the byte length of a file in the directory.
      protected void fsync​(java.lang.String name)  
      java.nio.file.Path getDirectory()  
      java.util.Set<java.lang.String> getPendingDeletions()
      Returns a set of files currently pending deletion in this directory.
      java.lang.String[] listAll()
      Returns names of all files stored in this directory.
      static java.lang.String[] listAll​(java.nio.file.Path dir)
      Lists all files (including subdirectories) in the directory.
      private static java.lang.String[] listAll​(java.nio.file.Path dir, java.util.Set<java.lang.String> skipNames)  
      private void maybeDeletePendingFiles()  
      static FSDirectory open​(java.nio.file.Path path)
      Creates an FSDirectory instance, trying to pick the best implementation given the current environment.
      static FSDirectory open​(java.nio.file.Path path, LockFactory lockFactory)
      Just like open(Path), but allows you to also specify a custom LockFactory.
      private void privateDeleteFile​(java.lang.String name, boolean isPendingDelete)  
      void rename​(java.lang.String source, java.lang.String dest)
      Renames source file to dest file where dest must not already exist in the directory.
      void sync​(java.util.Collection<java.lang.String> names)
      Ensures that any writes to these files are moved to stable storage (made durable).
      void syncMetaData()
      Ensures that directory metadata, such as recent file renames, are moved to stable storage.
      java.lang.String toString()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Field Detail

      • directory

        protected final java.nio.file.Path directory

      • pendingDeletes

        private final java.util.Set<java.lang.String> pendingDeletes
        Maps files that we are trying to delete (or we tried already but failed) before attempting to delete that key.

      • opsSinceLastDelete

        private final java.util.concurrent.atomic.AtomicInteger opsSinceLastDelete

    • Constructor Detail

      • FSDirectory

        protected FSDirectory​(java.nio.file.Path path,
                      LockFactory lockFactory)
               throws java.io.IOException
        Create a new FSDirectory for the named location (ctor for subclasses). The directory is created at the named location if it does not yet exist.

        FSDirectory resolves the given Path to a canonical / real path to ensure it can correctly lock the index directory and no other process can interfere with changing possible symlinks to the index directory inbetween. If you want to use symlinks and change them dynamically, close all IndexWriters and create a new FSDirectory instance.

        Parameters:
        path - the path of the directory
        lockFactory - the lock factory to use, or null for the default (NativeFSLockFactory);
        Throws:
        java.io.IOException - if there is a low-level I/O error

    • Method Detail

      • open

        public static FSDirectory open​(java.nio.file.Path path)
                        throws java.io.IOException
        Creates an FSDirectory instance, trying to pick the best implementation given the current environment. The directory returned uses the NativeFSLockFactory. The directory is created at the named location if it does not yet exist.

        FSDirectory resolves the given Path when calling this method to a canonical / real path to ensure it can correctly lock the index directory and no other process can interfere with changing possible symlinks to the index directory inbetween. If you want to use symlinks and change them dynamically, close all IndexWriters and create a new FSDirectory instance.

        Currently this returns MMapDirectory for Linux, MacOSX, Solaris, and Windows 64-bit JREs, NIOFSDirectory for other non-Windows JREs, and SimpleFSDirectory for other JREs on Windows. It is highly recommended that you consult the implementation's documentation for your platform before using this method.

        NOTE: this method may suddenly change which implementation is returned from release to release, in the event that higher performance defaults become possible; if the precise implementation is important to your application, please instantiate it directly, instead. For optimal performance you should consider using MMapDirectory on 64 bit JVMs.

        See above

        Throws:
        java.io.IOException

      • open

        public static FSDirectory open​(java.nio.file.Path path,
                               LockFactory lockFactory)
                        throws java.io.IOException
        Just like open(Path), but allows you to also specify a custom LockFactory.
        Throws:
        java.io.IOException

      • listAll

        public static java.lang.String[] listAll​(java.nio.file.Path dir)
                                  throws java.io.IOException
        Lists all files (including subdirectories) in the directory.
        Throws:
        java.io.IOException - if there was an I/O error during listing

      • listAll

        private static java.lang.String[] listAll​(java.nio.file.Path dir,
                                          java.util.Set<java.lang.String> skipNames)
                                   throws java.io.IOException
        Throws:
        java.io.IOException

      • listAll

        public java.lang.String[] listAll()
                           throws java.io.IOException
        Description copied from class: Directory
        Returns names of all files stored in this directory. The output must be in sorted (UTF-16, java's String.compareTo(java.lang.String)) order.
        Specified by:
        listAll in class Directory
        Throws:
        java.io.IOException - in case of I/O error

      • fileLength

        public long fileLength​(java.lang.String name)
                throws java.io.IOException
        Description copied from class: Directory
        Returns the byte length of a file in the directory. This method must throw either NoSuchFileException or FileNotFoundException if name points to a non-existing file.
        Specified by:
        fileLength in class Directory
        Parameters:
        name - the name of an existing file.
        Throws:
        java.io.IOException - in case of I/O error

      • createOutput

        public IndexOutput createOutput​(java.lang.String name,
                                IOContext context)
                         throws java.io.IOException
        Description copied from class: Directory
        Creates a new, empty file in the directory and returns an IndexOutput instance for appending data to this file. This method must throw FileAlreadyExistsException if the file already exists.
        Specified by:
        createOutput in class Directory
        Parameters:
        name - the name of the file to create.
        Throws:
        java.io.IOException - in case of I/O error

      • createTempOutput

        public IndexOutput createTempOutput​(java.lang.String prefix,
                                    java.lang.String suffix,
                                    IOContext context)
                             throws java.io.IOException
        Description copied from class: Directory
        Creates a new, empty, temporary file in the directory and returns an IndexOutput instance for appending data to this file. The temporary file name (accessible via IndexOutput.getName()) will start with prefix, end with suffix and have a reserved file extension .tmp.
        Specified by:
        createTempOutput in class Directory
        Throws:
        java.io.IOException

      • ensureCanRead

        protected void ensureCanRead​(java.lang.String name)
                      throws java.io.IOException
        Throws:
        java.io.IOException

      • sync

        public void sync​(java.util.Collection<java.lang.String> names)
          throws java.io.IOException
        Description copied from class: Directory
        Ensures that any writes to these files are moved to stable storage (made durable). Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.
        Specified by:
        sync in class Directory
        Throws:
        java.io.IOException
        See Also:
        Directory.syncMetaData()

      • rename

        public void rename​(java.lang.String source,
                   java.lang.String dest)
            throws java.io.IOException
        Description copied from class: Directory
        Renames source file to dest file where dest must not already exist in the directory. It is permitted for this operation to not be truly atomic, for example both source and dest can be visible temporarily in Directory.listAll(). However, the implementation of this method must ensure the content of dest appears as the entire source atomically. So once dest is visible for readers, the entire content of previous source is visible. This method is used by IndexWriter to publish commits.
        Specified by:
        rename in class Directory
        Throws:
        java.io.IOException

      • syncMetaData

        public void syncMetaData()
                  throws java.io.IOException
        Description copied from class: Directory
        Ensures that directory metadata, such as recent file renames, are moved to stable storage.
        Specified by:
        syncMetaData in class Directory
        Throws:
        java.io.IOException
        See Also:
        Directory.sync(Collection)

      • close

        public void close()
           throws java.io.IOException
        Description copied from class: Directory
        Closes the directory.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in class Directory
        Throws:
        java.io.IOException

      • getDirectory

        public java.nio.file.Path getDirectory()
        Returns:
        the underlying filesystem directory

      • fsync

        protected void fsync​(java.lang.String name)
              throws java.io.IOException
        Throws:
        java.io.IOException

      • deleteFile

        public void deleteFile​(java.lang.String name)
                throws java.io.IOException
        Description copied from class: Directory
        Removes an existing file in the directory. This method must throw either NoSuchFileException or FileNotFoundException if name points to a non-existing file.
        Specified by:
        deleteFile in class Directory
        Parameters:
        name - the name of an existing file.
        Throws:
        java.io.IOException - in case of I/O error

      • deletePendingFiles

        public void deletePendingFiles()
                        throws java.io.IOException
        Try to delete any pending files that we had previously tried to delete but failed because we are on Windows and the files were still held open.
        Throws:
        java.io.IOException

      • maybeDeletePendingFiles

        private void maybeDeletePendingFiles()
                              throws java.io.IOException
        Throws:
        java.io.IOException

      • privateDeleteFile

        private void privateDeleteFile​(java.lang.String name,
                               boolean isPendingDelete)
                        throws java.io.IOException
        Throws:
        java.io.IOException

      • getPendingDeletions

        public java.util.Set<java.lang.String> getPendingDeletions()
                                                    throws java.io.IOException
        Description copied from class: Directory
        Returns a set of files currently pending deletion in this directory.
        Specified by:
        getPendingDeletions in class Directory
        Throws:
        java.io.IOException