Class LuceneDataSearchEngine

java.lang.Object
com.jalios.jcms.search.LuceneDataSearchEngine
All Implemented Interfaces:
JcmsConstants, JaliosConstants
Direct Known Subclasses:
LuceneCategorySearchEngine, LuceneMemberSearchEngine, LucenePublicationSearchEngine

public abstract class LuceneDataSearchEngine extends Object implements JcmsConstants
Base class for indexing JCMS Data objects using Lucene.

Index layout

Each engine maintains two on-disk index instances named idx0 and idx1, located under the engine root directory:

   WEB-INF/data/lucene/{directoryName}/
     active          <-- persisted name of the currently active instance
     idx0/           <-- instance 0
       fr/           <--   language sub-directory (multilingual engines)
       en/
     idx1/           <-- instance 1 (built during a full reindex)
 
For monolingual engines the language sub-directory is omitted:
   WEB-INF/data/lucene/{directoryName}/
     active
     idx0/
     idx1/
 

Blue/Green full reindex

When a full reindex is requested, a new (pending) instance is built while the current (active) instance keeps serving searches without interruption. Once the build is complete an atomic swap makes the pending instance active. A short catchup reindex then replays any documents written to the former active instance during the build, blocking incremental writes only for its own duration.

Thread safety

  • Each LuceneDataSearchEngine.IndexInstance owns a ReentrantLock (writeLock) that serializes all write operations into that instance.
  • The consumer thread always acquires activeInstance.get().writeLock at the start of each batch, re-reading the reference after the lock is acquired to handle a concurrent swap transparently.
  • The active instance reference is an AtomicReference so that read-only searcher acquisition is lock-free.
  • Reference counting on each LuceneDataSearchEngine.IndexInstance ensures it is not closed while in-flight searches still hold a searcher against it.
Since:
jcms-5.5.0
  • Field Details

    • ID_FIELD

      public static final String ID_FIELD
      See Also:
    • INDEXING_DATE_FIELD

      public static final String INDEXING_DATE_FIELD
      See Also:
    • INDEXING_DATE_EXTRAINFO

      public static final String INDEXING_DATE_EXTRAINFO
      See Also:
    • BACKGROUND_PROCESS_MGR

      public static final BackgroundProcessManager BACKGROUND_PROCESS_MGR
    • channel

      protected final Channel channel
    • alarmMgr

      protected final AlarmManager alarmMgr
    • engineName

      protected final String engineName
    • directoryName

      protected final String directoryName
    • multilingual

      protected final boolean multilingual
    • langList

      protected final List<String> langList
      Languages indexed by this engine. All site languages for multilingual engines; default language only otherwise.
  • Constructor Details

    • LuceneDataSearchEngine

      protected LuceneDataSearchEngine(String engineName, String directoryName, boolean multilingual) throws Exception
      Initializes a new Lucene Data Search Engine.
      Parameters:
      engineName - human-readable engine name (e.g. "Publication", "Member")
      directoryName - on-disk directory name under the Lucene root (e.g. "PublicationsIndices", "MemberIndex")
      multilingual - true to maintain one index per site language, false to use only the default site language
      Throws:
      Exception - if Lucene resources cannot be initialized
  • Method Details

    • getDirectory

      public org.apache.lucene.store.FSDirectory getDirectory(String lang)
      Returns the FSDirectory used for the given language in the active instance. For read-only access only.
      Parameters:
      lang - ISO-639 language code
      Returns:
      the active FSDirectory, or null if unavailable
    • getIndexWriter

      public org.apache.lucene.index.IndexWriter getIndexWriter(String lang)
      Returns the IndexWriter used for the given language in the active instance.
      Parameters:
      lang - ISO-639 language code
      Returns:
      the active IndexWriter, or null if unavailable
    • getSearcherManager

      public org.apache.lucene.search.SearcherManager getSearcherManager(String lang)
      Returns the SearcherManager used for the given language in the active instance.
      Parameters:
      lang - ISO-639 language code
      Returns:
      the active SearcherManager, or null if unavailable
    • getAnalyzer

      public org.apache.lucene.analysis.Analyzer getAnalyzer(String lang, boolean isIndexing)
      Returns the Analyzer to use for the given language and operation type.
      Parameters:
      lang - ISO-639 language code, may be null
      isIndexing - true for indexing, false for querying
      Returns:
      an Analyzer instance, never null
    • getSimilarity

      public org.apache.lucene.search.similarities.Similarity getSimilarity()
      Returns the Similarity used by searchers of this engine.
      Returns:
      the Similarity instance, never null
    • createSnapshot

      public void createSnapshot(File targetDirectory) throws IOException
      Creates a point-in-time snapshot of the active index instance inside the specified target directory. Only the active instance is snapshotted; the pending instance (if any) is excluded.
      Parameters:
      targetDirectory - directory in which to store the snapshot, e.g. WEB-INF/data/lucene-snapshots/snapshot-2021-04-29-1640
      Throws:
      IOException - on any I/O error during the snapshot
      Since:
      jcms-10.0.5 / JCMS-8395
    • index

      protected void index(org.apache.lucene.index.IndexWriter writer, Collection<? extends Data> coll, String lang) throws IOException
      Indexes a collection of Data objects into the given writer for the specified language. The caller is responsible for synchronization.
      Parameters:
      writer - Lucene writer to use
      coll - collection of Data to index; must not be null
      lang - ISO-639 language code
      Throws:
      IOException - on I/O error
    • index

      protected void index(org.apache.lucene.index.IndexWriter writer, Iterator<? extends Data> iterator, int iteratorSize, String lang) throws IOException
      Indexes all Data objects returned by the given iterator into the specified writer for the given language. The caller is responsible for synchronization.
      Parameters:
      writer - Lucene writer to use
      iterator - iterator over Data to index; must not be null
      iteratorSize - total number of elements (used for progress only)
      lang - ISO-639 language code
      Throws:
      IOException - on I/O error
    • remove

      protected void remove(org.apache.lucene.index.IndexWriter writer, Collection<? extends Data> coll, String lang) throws IOException
      Removes a collection of Data objects from the index. The caller is responsible for synchronization.
      Parameters:
      writer - Lucene writer to use
      coll - collection of Data to remove; must not be null
      lang - ISO-639 language code
      Throws:
      IOException - on I/O error
    • getPrimaryTerm

      protected org.apache.lucene.index.Term getPrimaryTerm(Data data)
      Returns the Lucene Term used as the primary key for the given data when searching, updating, or deleting its index document.
      Parameters:
      data - the Data whose primary term is needed
      Returns:
      a non-null Term
    • acquireSearcher

      protected org.apache.lucene.search.IndexSearcher acquireSearcher(String[] languages, String defaultLanguage) throws IOException
      Acquires an IndexSearcher for the given languages. For a single-language search the searcher is obtained from the active instance's SearcherManager and wrapped in an LuceneDataSearchEngine.InstanceAwareSearcher that carries a reference count, preventing the instance from being closed while the search is in progress. For multi-language searches a MultiReader is used, backed by a per-instance cache keyed by the comma-separated language list. The returned searcher must be passed to releaseSearcher(java.lang.String[], java.lang.String, org.apache.lucene.search.IndexSearcher) after use.
      Parameters:
      languages - requested ISO-639 language codes; null or empty means "use the current user's language or all languages"
      defaultLanguage - fallback language if no explicit language is given and the current user language is unavailable
      Returns:
      an IndexSearcher (actually an LuceneDataSearchEngine.InstanceAwareSearcher)
      Throws:
      IOException - on I/O error
    • releaseSearcher

      protected void releaseSearcher(String[] languages, String defaultLanguage, org.apache.lucene.search.IndexSearcher searcher)
      Releases an IndexSearcher previously obtained via acquireSearcher(java.lang.String[], java.lang.String). Decrements the source LuceneDataSearchEngine.IndexInstance's reference count; if the instance has been discarded and the count reaches zero its resources are closed and its on-disk directories are deleted asynchronously.
      Parameters:
      languages - same value passed to acquireSearcher(java.lang.String[], java.lang.String)
      defaultLanguage - same value passed to acquireSearcher(java.lang.String[], java.lang.String)
      searcher - the searcher to release; must be an LuceneDataSearchEngine.InstanceAwareSearcher
    • clearIndices

      protected void clearIndices()
      Deletes all documents from all language indices of the active instance. This operation is irreversible. Blocks the consumer thread for its entire duration.
    • optimizeIndices

      public void optimizeIndices()
      Optimizes (force-merges) all language indices of the active instance. This is a potentially long and resource-intensive operation. It is rejected silently if a full reindex is currently in progress.
    • reindexAll

      public void reindexAll() throws IOException
      Triggers a full Blue/Green reindex of all data. Delegates to reindex(ReindexOptions).
      Throws:
      IOException - on I/O error
    • getReindexAllAction

      protected abstract AdminSearchEnginesHandler.Action getReindexAllAction()
      Returns the AdminSearchEnginesHandler.Action to use when starting an AdminSearchEngineProcess for a full reindex.
      Since:
      10.0.8 / JCMS-10687
    • reindex

      public void reindex(LuceneDataSearchEngine.ReindexOptions options) throws IOException
      Reindexes data according to the given options.
      • If options.clearIndicesFirst is true, a Blue/Green full reindex is performed: a new index instance is built while the current one keeps serving searches, then an atomic swap is performed and a catchup reindex runs asynchronously.
      • Otherwise, a partial reindex of documents modified after options.modifiedAfterDate is performed directly on the active instance under its writeLock.
      A concurrent full reindex request is rejected (logged as a warning).
      Parameters:
      options - reindex configuration; must not be null
      Throws:
      IOException - on I/O error
    • getLuceneDocument

      public org.apache.lucene.document.Document getLuceneDocument(Data data, String lang)
      Returns the Lucene Document corresponding to the given Data in the index of the specified language, or null if not found.
      Parameters:
      data - the Data to look up
      lang - ISO-639 language code
      Returns:
      a Lucene Document, or null
      Since:
      jcms-6.0.1
    • getIndexingDate

      public Date getIndexingDate(Data data, String lang)
      Returns the date at which the given Data was last indexed for the given language, or null if it has not been indexed.
      Parameters:
      data - the Data to check
      lang - ISO-639 language code
      Returns:
      the indexing Date, or null
      Since:
      jcms-6.0.1
    • getIndexingDate

      public Date getIndexingDate(Data data)
      Returns the date at which the given Data was last indexed in the site's default language. The result is cached in the data's extra-info map.
      Parameters:
      data - the Data to check
      Returns:
      the indexing Date, or null
      Since:
      jcms-6.0.1
    • getSortFieldName

      protected String getSortFieldName(String rawFieldName)
      Returns the Lucene field name to use for sort operations on the given field.
      Parameters:
      rawFieldName - base field name (e.g. "title", "firstname")
      Returns:
      sort field name (e.g. "title_sort", "firstname_sort")
      Since:
      jcms-11.0.0 / JCMS-11802
    • addData

      protected void addData(Data data)
      Enqueues the given Data for asynchronous addition to the index.
      Parameters:
      data - the Data to add; null is silently ignored
    • updateData

      protected void updateData(Data data)
      Enqueues the given Data for asynchronous update in the index.
      Parameters:
      data - the Data to update; null is silently ignored
    • deleteData

      protected void deleteData(Data data)
      Enqueues the given Data for asynchronous deletion from the index.
      Parameters:
      data - the Data to delete; null is silently ignored
    • addDataCollection

      protected void addDataCollection(Collection<? extends Data> coll)
      Enqueues a collection of Data objects for asynchronous addition.
      Parameters:
      coll - the collection to add
    • updateDataCollection

      protected void updateDataCollection(Collection<? extends Data> coll)
      Enqueues a collection of Data objects for asynchronous update.
      Parameters:
      coll - the collection to update
    • deleteDataCollection

      protected void deleteDataCollection(Collection<? extends Data> coll)
      Enqueues a collection of Data objects for asynchronous deletion.
      Parameters:
      coll - the collection to delete
    • isOperationRunning

      public boolean isOperationRunning()
      Returns true if a reindex or optimize operation is currently running.
      See Also:
    • getProgressState

      public int getProgressState()
      Returns the current operation progress as a percentage (0–100), or 100 if no operation is running.
      See Also:
    • getOperationStartTime

      public long getOperationStartTime()
      Returns the start time of the current operation in epoch milliseconds, or 0 if no operation is running.
      See Also:
    • getLastReindexDateSinceRestart

      public Date getLastReindexDateSinceRestart()
      Returns the date of the last reindex since the last JVM restart, or null if no reindex has been performed.
    • getLastReindexDuration

      public long getLastReindexDuration()
      Returns the duration of the last reindex in milliseconds, or 0 if none has been performed since the last JVM restart.
    • getLastOptimizeDateSinceRestart

      public Date getLastOptimizeDateSinceRestart()
      Returns the date of the last optimize since the last JVM restart, or null if no optimize has been performed.
    • getLastOptimizeDuration

      public long getLastOptimizeDuration()
      Returns the duration of the last optimize in milliseconds, or 0 if none has been performed since the last JVM restart.
    • getIndexingLatch

      public LuceneDataSearchEngine.IndexingLatch getIndexingLatch(Data data)
      Returns a new LuceneDataSearchEngine.IndexingLatch that will be released after the next indexing operation on the given Data completes.
      Parameters:
      data - the Data to monitor; null returns null
      Returns:
      a new LuceneDataSearchEngine.IndexingLatch, or null if data is null
      Since:
      jcms-8.0.2 / JCMS-3805
    • getLogger

      protected abstract org.apache.log4j.Logger getLogger()
      Returns the Logger used for all log messages emitted by this engine.
      Returns:
      a non-null Logger
    • getDataIterator

      protected abstract DataIterator<Data> getDataIterator(LuceneDataSearchEngine.ReindexOptions options)
      Returns a DataIterator over all Data objects to be indexed, filtered according to the given LuceneDataSearchEngine.ReindexOptions.
      Parameters:
      options - reindex configuration
      Returns:
      a non-null DataIterator
      Since:
      jcms-10.0.5 / JCMS-8170
    • indexData

      protected abstract void indexData(org.apache.lucene.index.IndexWriter writer, Data data, String lang, boolean isFullReindexing) throws IOException
      Indexes a single Data object into the given IndexWriter for the specified language.
      Parameters:
      writer - target Lucene writer
      data - the Data to index
      lang - ISO-639 language code
      isFullReindexing - true if operation is for a full reindexing with a empty index, false if updating an existing index
      Throws:
      IOException - on I/O error