Package com.jalios.jcms.search
Class LuceneDataSearchEngine
java.lang.Object
com.jalios.jcms.search.LuceneDataSearchEngine
- All Implemented Interfaces:
JcmsConstants,JaliosConstants
- Direct Known Subclasses:
LuceneCategorySearchEngine,LuceneMemberSearchEngine,LucenePublicationSearchEngine
Base class for indexing JCMS
Data objects using Lucene.
Index layout
Each engine maintains two on-disk index instances named idx0 and
idx1, located under the engine root directory:
WEB-INF/data/lucene/{directoryName}/
active <-- persisted name of the currently active instance
idx0/ <-- instance 0
fr/ <-- language sub-directory (multilingual engines)
en/
idx1/ <-- instance 1 (built during a full reindex)
For monolingual engines the language sub-directory is omitted:
WEB-INF/data/lucene/{directoryName}/
active
idx0/
idx1/
Blue/Green full reindex
When a full reindex is requested, a new (pending) instance is built while the current (active) instance keeps serving searches without interruption. Once the build is complete an atomic swap makes the pending instance active. A short catchup reindex then replays any documents written to the former active instance during the build, blocking incremental writes only for its own duration.
Thread safety
- Each
LuceneDataSearchEngine.IndexInstanceowns aReentrantLock(writeLock) that serializes all write operations into that instance. - The consumer thread always acquires
activeInstance.get().writeLockat the start of each batch, re-reading the reference after the lock is acquired to handle a concurrent swap transparently. - The active instance reference is an
AtomicReferenceso that read-only searcher acquisition is lock-free. - Reference counting on each
LuceneDataSearchEngine.IndexInstanceensures it is not closed while in-flight searches still hold a searcher against it.
- Since:
- jcms-5.5.0
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic classACountDownLatchthat reaches zero when the next indexing operation on a specificDatacompletes.static classEncapsulates options controlling a reindex operation. -
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final AlarmManagerstatic final BackgroundProcessManagerprotected final Channelprotected final Stringprotected final Stringstatic final Stringstatic final Stringstatic final StringLanguages indexed by this engine.protected final booleanFields inherited from interface com.jalios.util.JaliosConstants
CRLF, MILLIS_IN_ONE_DAY, MILLIS_IN_ONE_HOUR, MILLIS_IN_ONE_MINUTE, MILLIS_IN_ONE_MONTH, MILLIS_IN_ONE_SECOND, MILLIS_IN_ONE_WEEK, MILLIS_IN_ONE_YEARFields inherited from interface com.jalios.jcms.JcmsConstants
ADATE_SEARCH, ADVANCED_TAB, AJAX_MODE_ATTR, ARCHIVES_DIR, ASCII_WIDTH, CATEGORY_TAB, CDATE_SEARCH, CLASS_PROPERTY, COMMON_ALARM, CONTENT_TAB, COOKIE_MAX_AGE, COUNTRY_SPRITE, CS_TYPOLOGY_ROOT_CAT_VID, CTRL_TOPIC_INTERNAL, CTRL_TOPIC_REF, CTRL_TOPIC_VALUE, CTRL_TOPIC_WRITE, CUSTOM_PROP, DATA_DIRECTORY, DEFAULT_PHOTO_PROP, DOCCHOOSER_HEIGHT, DOCCHOOSER_WIDTH, DOCS_DIR, EDATE_SEARCH, EMAIL_REGEXP, ERROR_MSG, FORBIDDEN_FILE_ACCESS, FORBIDDEN_REDIRECT, FORCE_REDIRECT, GENERATIONAL_STORE_DIRECTORY, GLYPH_ICON_PREFIX, ICON_ARCHIVE, ICON_ICON_PREFIX, ICON_LOCK, ICON_LOCK_STRONG, ICON_PREFIX_PROP, ICON_WARN, ICON_WH_BOOK_CLOSED, ICON_WH_BOOK_OPEN, INFORMATION_MSG, IS_IN_FRONT_OFFICE, JALIOS_JUNIT_PROP, JCMS_CADDY, JCMS_MSG_LIST, JCMS_TOASTR_COLLECTION, JSYNC_DOWNLOAD_DIR, JSYNC_SYNC_ALARM, LANG_SPRITE, LOG_FILE, LOG_TOPIC_SECURITY, LOGGER_PROP, LOGGER_XMLPROP, MBR_PHOTO_DIR, MDATE_SEARCH, MONITOR_XML, OP_CREATE, OP_CREATE_STR, OP_DEEP_COPY, OP_DEEP_COPY_STR, OP_DEEP_DELETE, OP_DEEP_DELETE_STR, OP_DELETE, OP_DELETE_STR, OP_MERGE, OP_MERGE_STR, OP_UPDATE, OP_UPDATE_STR, ORGANIZATION_ROOT_GROUP_PROP, PDATE_SEARCH, PHOTO_DIR, PHOTO_ICON, PHOTO_ICON_HEIGHT, PHOTO_ICON_PROP_PREFIX, PHOTO_ICON_WIDTH, PHOTO_LARGE, PHOTO_LARGE_HEIGHT, PHOTO_LARGE_PROP_PREFIX, PHOTO_LARGE_WIDTH, PHOTO_MINI, PHOTO_MINI_HEIGHT, PHOTO_MINI_PROP_PREFIX, PHOTO_MINI_WIDTH, PHOTO_NORMAL, PHOTO_NORMAL_HEIGHT, PHOTO_NORMAL_PROP_PREFIX, PHOTO_NORMAL_WIDTH, PHOTO_SMALL, PHOTO_SMALL_HEIGHT, PHOTO_SMALL_PROP_PREFIX, PHOTO_SMALL_WIDTH, PHOTO_TINY, PHOTO_TINY_HEIGHT, PHOTO_TINY_PROP_PREFIX, PHOTO_TINY_WIDTH, PHOTO_XLARGE, PHOTO_XLARGE_HEIGHT, PHOTO_XLARGE_PROP_PREFIX, PHOTO_XLARGE_WIDTH, PREVIOUS_TAB, PRINT_VIEW, PRIVATE_FILE_ACCESS, PUBLIC_FILE_ACCESS, RAW_CONTENT_ICON_PREFIX, READ_RIGHT_TAB, SDATE_SEARCH, SEARCHENGINE_ALARM, SECURITY_LOG_FILE, SESSION_AUTHORIZED_FILENAMES_SET, SPRITE_ICON_PREFIX, STATS_REPORT_DIR, STATUS_PROP, STORE_XML, SUCCESS_MSG, SVG_ICON_PREFIX, SVGINLINE_ICON_PREFIX, SVGSPRITE_ICON_PREFIX, TEMPLATE_TAB, THEME_DIR, THUMBNAIL_LARGE_HEIGHT, THUMBNAIL_LARGE_WIDTH, THUMBNAIL_SMALL_HEIGHT, THUMBNAIL_SMALL_WIDTH, TTCARD_MEDIA_HEIGHT, TTCARD_MEDIA_WIDTH, TYPES_ICON_ALT_PROP, TYPES_ICON_SUFFIX_PROP, TYPES_ICON_TITLE_PROP, TYPES_PREFIX_PROP, TYPES_THUMB_SUFFIX_PROP, TYPES_UNKNOWN_CLASS_PROP, UDATE_SEARCH, UPDATE_RIGHT_TAB, UPLOAD_DIR, UPLOAD_PERMISSION_COUNT_PROP_PREFIX, UPLOAD_PERMISSION_SIZE_PROP_PREFIX, URL_REGEXP, VID_LOGGED_MEMBER, WARNING_MSG, WEBAPP_PROP, WFEXPRESS_ALARM, WFREMINDER_ALARM, WORKFLOW_TAB, WORKFLOW_XML -
Constructor Summary
ConstructorsModifierConstructorDescriptionprotectedLuceneDataSearchEngine(String engineName, String directoryName, boolean multilingual) Initializes a new Lucene Data Search Engine. -
Method Summary
Modifier and TypeMethodDescriptionprotected org.apache.lucene.search.IndexSearcheracquireSearcher(String[] languages, String defaultLanguage) Acquires anIndexSearcherfor the given languages.protected voidEnqueues the givenDatafor asynchronous addition to the index.protected voidaddDataCollection(Collection<? extends Data> coll) Enqueues a collection ofDataobjects for asynchronous addition.protected voidDeletes all documents from all language indices of the active instance.voidcreateSnapshot(File targetDirectory) Creates a point-in-time snapshot of the active index instance inside the specified target directory.protected voiddeleteData(Data data) Enqueues the givenDatafor asynchronous deletion from the index.protected voiddeleteDataCollection(Collection<? extends Data> coll) Enqueues a collection ofDataobjects for asynchronous deletion.org.apache.lucene.analysis.AnalyzergetAnalyzer(String lang, boolean isIndexing) Returns theAnalyzerto use for the given language and operation type.protected abstract DataIterator<Data> Returns aDataIteratorover allDataobjects to be indexed, filtered according to the givenLuceneDataSearchEngine.ReindexOptions.org.apache.lucene.store.FSDirectorygetDirectory(String lang) Returns theFSDirectoryused for the given language in the active instance.getIndexingDate(Data data) Returns the date at which the givenDatawas last indexed in the site's default language.getIndexingDate(Data data, String lang) Returns the date at which the givenDatawas last indexed for the given language, ornullif it has not been indexed.getIndexingLatch(Data data) Returns a newLuceneDataSearchEngine.IndexingLatchthat will be released after the next indexing operation on the givenDatacompletes.org.apache.lucene.index.IndexWritergetIndexWriter(String lang) Returns theIndexWriterused for the given language in the active instance.Returns the date of the last optimize since the last JVM restart, ornullif no optimize has been performed.longReturns the duration of the last optimize in milliseconds, or 0 if none has been performed since the last JVM restart.Returns the date of the last reindex since the last JVM restart, ornullif no reindex has been performed.longReturns the duration of the last reindex in milliseconds, or 0 if none has been performed since the last JVM restart.protected abstract org.apache.log4j.LoggerReturns theLoggerused for all log messages emitted by this engine.org.apache.lucene.document.DocumentgetLuceneDocument(Data data, String lang) Returns the LuceneDocumentcorresponding to the givenDatain the index of the specified language, ornullif not found.longReturns the start time of the current operation in epoch milliseconds, or 0 if no operation is running.protected org.apache.lucene.index.TermgetPrimaryTerm(Data data) Returns the LuceneTermused as the primary key for the given data when searching, updating, or deleting its index document.intReturns the current operation progress as a percentage (0–100), or 100 if no operation is running.protected abstract AdminSearchEnginesHandler.ActionReturns theAdminSearchEnginesHandler.Actionto use when starting anAdminSearchEngineProcessfor a full reindex.org.apache.lucene.search.SearcherManagergetSearcherManager(String lang) Returns theSearcherManagerused for the given language in the active instance.org.apache.lucene.search.similarities.SimilarityReturns theSimilarityused by searchers of this engine.protected StringgetSortFieldName(String rawFieldName) Returns the Lucene field name to use for sort operations on the given field.protected voidindex(org.apache.lucene.index.IndexWriter writer, Collection<? extends Data> coll, String lang) Indexes a collection ofDataobjects into the given writer for the specified language.protected voidindex(org.apache.lucene.index.IndexWriter writer, Iterator<? extends Data> iterator, int iteratorSize, String lang) Indexes allDataobjects returned by the given iterator into the specified writer for the given language.protected abstract voidindexData(org.apache.lucene.index.IndexWriter writer, Data data, String lang, boolean isFullReindexing) Indexes a singleDataobject into the givenIndexWriterfor the specified language.booleanReturnstrueif a reindex or optimize operation is currently running.voidOptimizes (force-merges) all language indices of the active instance.voidReindexes data according to the given options.voidTriggers a full Blue/Green reindex of all data.protected voidreleaseSearcher(String[] languages, String defaultLanguage, org.apache.lucene.search.IndexSearcher searcher) Releases anIndexSearcherpreviously obtained viaacquireSearcher(java.lang.String[], java.lang.String).protected voidremove(org.apache.lucene.index.IndexWriter writer, Collection<? extends Data> coll, String lang) Removes a collection ofDataobjects from the index.protected voidupdateData(Data data) Enqueues the givenDatafor asynchronous update in the index.protected voidupdateDataCollection(Collection<? extends Data> coll) Enqueues a collection ofDataobjects for asynchronous update.
-
Field Details
-
ID_FIELD
- See Also:
-
INDEXING_DATE_FIELD
- See Also:
-
INDEXING_DATE_EXTRAINFO
- See Also:
-
BACKGROUND_PROCESS_MGR
-
channel
-
alarmMgr
-
engineName
-
directoryName
-
multilingual
protected final boolean multilingual -
langList
Languages indexed by this engine. All site languages for multilingual engines; default language only otherwise.
-
-
Constructor Details
-
LuceneDataSearchEngine
protected LuceneDataSearchEngine(String engineName, String directoryName, boolean multilingual) throws Exception Initializes a new Lucene Data Search Engine.- Parameters:
engineName- human-readable engine name (e.g. "Publication", "Member")directoryName- on-disk directory name under the Lucene root (e.g. "PublicationsIndices", "MemberIndex")multilingual-trueto maintain one index per site language,falseto use only the default site language- Throws:
Exception- if Lucene resources cannot be initialized
-
-
Method Details
-
getDirectory
Returns theFSDirectoryused for the given language in the active instance. For read-only access only.- Parameters:
lang- ISO-639 language code- Returns:
- the active
FSDirectory, ornullif unavailable
-
getIndexWriter
Returns theIndexWriterused for the given language in the active instance.- Parameters:
lang- ISO-639 language code- Returns:
- the active
IndexWriter, ornullif unavailable
-
getSearcherManager
Returns theSearcherManagerused for the given language in the active instance.- Parameters:
lang- ISO-639 language code- Returns:
- the active
SearcherManager, ornullif unavailable
-
getAnalyzer
Returns theAnalyzerto use for the given language and operation type.- Parameters:
lang- ISO-639 language code, may benullisIndexing-truefor indexing,falsefor querying- Returns:
- an
Analyzerinstance, nevernull
-
getSimilarity
public org.apache.lucene.search.similarities.Similarity getSimilarity()Returns theSimilarityused by searchers of this engine.- Returns:
- the
Similarityinstance, nevernull
-
createSnapshot
Creates a point-in-time snapshot of the active index instance inside the specified target directory. Only the active instance is snapshotted; the pending instance (if any) is excluded.- Parameters:
targetDirectory- directory in which to store the snapshot, e.g.WEB-INF/data/lucene-snapshots/snapshot-2021-04-29-1640- Throws:
IOException- on any I/O error during the snapshot- Since:
- jcms-10.0.5 / JCMS-8395
-
index
protected void index(org.apache.lucene.index.IndexWriter writer, Collection<? extends Data> coll, String lang) throws IOException Indexes a collection ofDataobjects into the given writer for the specified language. The caller is responsible for synchronization.- Parameters:
writer- Lucene writer to usecoll- collection ofDatato index; must not benulllang- ISO-639 language code- Throws:
IOException- on I/O error
-
index
protected void index(org.apache.lucene.index.IndexWriter writer, Iterator<? extends Data> iterator, int iteratorSize, String lang) throws IOException Indexes allDataobjects returned by the given iterator into the specified writer for the given language. The caller is responsible for synchronization.- Parameters:
writer- Lucene writer to useiterator- iterator overDatato index; must not benulliteratorSize- total number of elements (used for progress only)lang- ISO-639 language code- Throws:
IOException- on I/O error
-
remove
protected void remove(org.apache.lucene.index.IndexWriter writer, Collection<? extends Data> coll, String lang) throws IOException Removes a collection ofDataobjects from the index. The caller is responsible for synchronization.- Parameters:
writer- Lucene writer to usecoll- collection ofDatato remove; must not benulllang- ISO-639 language code- Throws:
IOException- on I/O error
-
getPrimaryTerm
Returns the LuceneTermused as the primary key for the given data when searching, updating, or deleting its index document.- Parameters:
data- theDatawhose primary term is needed- Returns:
- a non-null
Term
-
acquireSearcher
protected org.apache.lucene.search.IndexSearcher acquireSearcher(String[] languages, String defaultLanguage) throws IOException Acquires anIndexSearcherfor the given languages. For a single-language search the searcher is obtained from the active instance'sSearcherManagerand wrapped in anLuceneDataSearchEngine.InstanceAwareSearcherthat carries a reference count, preventing the instance from being closed while the search is in progress. For multi-language searches aMultiReaderis used, backed by a per-instance cache keyed by the comma-separated language list. The returned searcher must be passed toreleaseSearcher(java.lang.String[], java.lang.String, org.apache.lucene.search.IndexSearcher)after use.- Parameters:
languages- requested ISO-639 language codes;nullor empty means "use the current user's language or all languages"defaultLanguage- fallback language if no explicit language is given and the current user language is unavailable- Returns:
- an
IndexSearcher(actually anLuceneDataSearchEngine.InstanceAwareSearcher) - Throws:
IOException- on I/O error
-
releaseSearcher
protected void releaseSearcher(String[] languages, String defaultLanguage, org.apache.lucene.search.IndexSearcher searcher) Releases anIndexSearcherpreviously obtained viaacquireSearcher(java.lang.String[], java.lang.String). Decrements the sourceLuceneDataSearchEngine.IndexInstance's reference count; if the instance has been discarded and the count reaches zero its resources are closed and its on-disk directories are deleted asynchronously.- Parameters:
languages- same value passed toacquireSearcher(java.lang.String[], java.lang.String)defaultLanguage- same value passed toacquireSearcher(java.lang.String[], java.lang.String)searcher- the searcher to release; must be anLuceneDataSearchEngine.InstanceAwareSearcher
-
clearIndices
protected void clearIndices()Deletes all documents from all language indices of the active instance. This operation is irreversible. Blocks the consumer thread for its entire duration. -
optimizeIndices
public void optimizeIndices()Optimizes (force-merges) all language indices of the active instance. This is a potentially long and resource-intensive operation. It is rejected silently if a full reindex is currently in progress. -
reindexAll
Triggers a full Blue/Green reindex of all data. Delegates toreindex(ReindexOptions).- Throws:
IOException- on I/O error
-
getReindexAllAction
Returns theAdminSearchEnginesHandler.Actionto use when starting anAdminSearchEngineProcessfor a full reindex.- Since:
- 10.0.8 / JCMS-10687
-
reindex
Reindexes data according to the given options.- If
options.clearIndicesFirstistrue, a Blue/Green full reindex is performed: a new index instance is built while the current one keeps serving searches, then an atomic swap is performed and a catchup reindex runs asynchronously. - Otherwise, a partial reindex of documents modified after
options.modifiedAfterDateis performed directly on the active instance under itswriteLock.
- Parameters:
options- reindex configuration; must not benull- Throws:
IOException- on I/O error
- If
-
getLuceneDocument
Returns the LuceneDocumentcorresponding to the givenDatain the index of the specified language, ornullif not found.- Parameters:
data- theDatato look uplang- ISO-639 language code- Returns:
- a Lucene
Document, ornull - Since:
- jcms-6.0.1
-
getIndexingDate
Returns the date at which the givenDatawas last indexed for the given language, ornullif it has not been indexed. -
getIndexingDate
Returns the date at which the givenDatawas last indexed in the site's default language. The result is cached in the data's extra-info map. -
getSortFieldName
Returns the Lucene field name to use for sort operations on the given field.- Parameters:
rawFieldName- base field name (e.g."title","firstname")- Returns:
- sort field name (e.g.
"title_sort","firstname_sort") - Since:
- jcms-11.0.0 / JCMS-11802
-
addData
Enqueues the givenDatafor asynchronous addition to the index.- Parameters:
data- theDatato add;nullis silently ignored
-
updateData
Enqueues the givenDatafor asynchronous update in the index.- Parameters:
data- theDatato update;nullis silently ignored
-
deleteData
Enqueues the givenDatafor asynchronous deletion from the index.- Parameters:
data- theDatato delete;nullis silently ignored
-
addDataCollection
Enqueues a collection ofDataobjects for asynchronous addition.- Parameters:
coll- the collection to add
-
updateDataCollection
Enqueues a collection ofDataobjects for asynchronous update.- Parameters:
coll- the collection to update
-
deleteDataCollection
Enqueues a collection ofDataobjects for asynchronous deletion.- Parameters:
coll- the collection to delete
-
isOperationRunning
public boolean isOperationRunning()Returnstrueif a reindex or optimize operation is currently running.- See Also:
-
getProgressState
public int getProgressState()Returns the current operation progress as a percentage (0–100), or 100 if no operation is running.- See Also:
-
getOperationStartTime
public long getOperationStartTime()Returns the start time of the current operation in epoch milliseconds, or 0 if no operation is running.- See Also:
-
getLastReindexDateSinceRestart
Returns the date of the last reindex since the last JVM restart, ornullif no reindex has been performed. -
getLastReindexDuration
public long getLastReindexDuration()Returns the duration of the last reindex in milliseconds, or 0 if none has been performed since the last JVM restart. -
getLastOptimizeDateSinceRestart
Returns the date of the last optimize since the last JVM restart, ornullif no optimize has been performed. -
getLastOptimizeDuration
public long getLastOptimizeDuration()Returns the duration of the last optimize in milliseconds, or 0 if none has been performed since the last JVM restart. -
getIndexingLatch
Returns a newLuceneDataSearchEngine.IndexingLatchthat will be released after the next indexing operation on the givenDatacompletes.- Parameters:
data- theDatato monitor;nullreturnsnull- Returns:
- a new
LuceneDataSearchEngine.IndexingLatch, ornullifdatais null - Since:
- jcms-8.0.2 / JCMS-3805
-
getLogger
protected abstract org.apache.log4j.Logger getLogger()Returns theLoggerused for all log messages emitted by this engine.- Returns:
- a non-null
Logger
-
getDataIterator
protected abstract DataIterator<Data> getDataIterator(LuceneDataSearchEngine.ReindexOptions options) Returns aDataIteratorover allDataobjects to be indexed, filtered according to the givenLuceneDataSearchEngine.ReindexOptions.- Parameters:
options- reindex configuration- Returns:
- a non-null
DataIterator - Since:
- jcms-10.0.5 / JCMS-8170
-
indexData
protected abstract void indexData(org.apache.lucene.index.IndexWriter writer, Data data, String lang, boolean isFullReindexing) throws IOException Indexes a singleDataobject into the givenIndexWriterfor the specified language.- Parameters:
writer- target Lucene writerdata- theDatato indexlang- ISO-639 language codeisFullReindexing- true if operation is for a full reindexing with a empty index, false if updating an existing index- Throws:
IOException- on I/O error
-