Abstract
Sometimes it is necessary to re-index something in the vault. For example, if the indexing parameters change or if there has been some problem in the indexing. Since we do not have exact information about the correspondents of the index when compared to the "real word", we cannot re-index just the erroneous objects. That is why we must re-index a larger batch or even all documents in the vault. That might be especially challenging when talking about production environment where the search must be operational all the time and the system performance cannot be affected too much because of re-indexing. This document describes different methods to re-index vault, also in a way that the search operations work during re-indexing.
NOTE 1! This document is valid for M-Files 20.9.x and beyond and IDOL version 12.x and beyond.
NOTE 2! Using this document requires basic understanding of IDOL components and terminology.
1. Initial remarks
- Currently, IDOL only uses so called single-pipeline indexing, which means that both meta- and filedata are indexed at the same time. That means, that triggering re-indexing always does full re-index (whichever method "just metadata or both" you choose), which might take much more time than just metadata.
- Indexing is done in batches of 10000 objects. If the whole vault is being re-indexed and you edit a file, it will be indexed in the next batch. Even though, it might take more time for the object changes to became searchable.
- If the environment resources (M-Files server CPUs and IDOL side disk I/O) allow, you can add more indexers for the re-indexing. When it is done, change the setting back to single indexer.
- Indexing English language vault with 4 indexers will usually index about 1 million documents in 24 hours.
- How to see if index rebuild is finished: /article/IDOL-How-to-see-if-index-rebuild-is-finished
2. Re-index method 1: Full, destructive
This method is the easiest one to trigger, but it will first destroy the whole index and then start rebuilding it, meaning the quick search will not work until material has been re-indexed. That is why it is not a recommended way to re-index at least a production environment.
1. Open the M-Files Admin tool.
2. Select 'Index Management' from the vault tree.
3. Select the index you want to rebuild.
4. Select 'Delete & Reindex'.
5. Click OK to the warning about long lasting indexing. Index rebuilding starts.
3. Re-index method 2: Full, non-destructive
This method just re-writes every object's index data, so the index will stay operational during the re-indexing.
1. Open the admin tool.
2. Select 'Index Management' from the vault tree.
3. Select the index you want to reindex.
4. Select 'Reindex'.
5. Click OK to the warning about long lasting indexing. Re-indexing starts.
4. Re-index method 3: Partial, non-destructive
Sometimes its preferred to re-index just a part of the index e.g., in the situation, where M-Files server upgrade has failed, and objects modified after the upgrade have been indexed with wrong parameters. Since we are using incremental timestamps, we always start from one point of time and move until the end of the objects. If you are using multiple indexers, note that you must change the timestamp for each indexer. Because we dint know exact timestamp for each indexer, use slightly lower timestamp that the lowest in the indexers if the wanted timestamp is not already lower. Then you can use that for all the indexers.
Using this method will need the usage of export scripts and the knowledge of index database name.
1. Search object and its ID you want to use as a starting point for indexing.
2. Use following SQL query to M-Files database to get the Timestamp of the corresponding object:
SELECT VERSIONFORMDI, ORIGINALOBJECTTYPE FROM [vault database name].[dbo].[OBJECTTYPEITEM] where [ID] = ID of the document
Example for database name Sample Vault SQL IDOL 12.4 and ID 230098:
SELECT VERSIONFORMDI, ORIGINALOBJECTTYPE FROM [Sample Vault SQL IDOL 12.4].[dbo].[OBJECTTYPEITEM] where [ID] = 230098
You will get following like table:
1088 0
0 17
189 101
168 149
The first column is the timestamp, the second is the Object Type. Object Type 0 is Document so usually we are interested of that. Get the timestamp value (1088 in the example table).
3. Stop MFIndexingManager service from M-Files server.
4. Get the database GUID from admin tool. Configurations > Advanced Vault Settings > Configuration > Search > Indexes > Index GUID e.g. C_... Select Advanced tab and from JSON copy the value of generatedIndexName or PredefinedIndexName.
5. We are using IndexInfo.ps1 Power Shell script and MFIDOLConsole.exe that are in IDOL installation package (\IDOL12_SetupTools_20200402_full_with_pwd\Installer\Shared\Scripts and Console). Copy those to IDOL frontend server.
6. Open Power Shell command prompt and go into Scripts folder.
7. Run following command. It will create an .idx file into ..\Scripts\Export\[datecode] folder:
powershell -nologo -noprofile -executionpolicy bypass -file .\IndexInfo.ps1 `
-dailyHost [IDOL hostname, IP address or just localhost if the script runs in IDOL server] `
-dailyACIPort [Daily ACI port] `
-idxRootFolderPath ".\export" `
-export `
-indexdb "[database GUID]"
Example:
powershell -nologo -noprofile -executionpolicy bypass -file .\IndexInfo.ps1 `
-dailyHost localhost `
-dailyACIPort 9000 `
-idxRootFolderPath ".\export" `
-export `
-indexdb "MF-{17E2AF7E-D782-4573-84EF-94CAE7CD30A0}"
8. Open .idx file in text editor.
9. Find object reference ending …TimestampCombinedV6-0. Each object starts with #DREREFERENCE and ends with #DREENDDOC.
Example:
#DREREFERENCE MF-{17E2AF7E-D782-4573-84EF-94CAE7CD30A0}_INTERNAL-InternalData-IndexInfo-TimestampCombinedV6-0
10. Change first value to the one you got in step 2. In example below, the value that should be changed is in red:
…{"values":["1088","132531025288200000","0","0"]}}}…
11. Remove all other objects from the file and save it.
12. Use IndexInfo.ps1 to import the file:
powershell -nologo -noprofile -executionpolicy bypass -file .\IndexInfo.ps1 `
-dailyHost [IDOL hostname, IP address or just localhost if the script runs in IDOL server] `
-dailyACIPort [Daily ACI port] `
-idxRootFolderPath ".\export" `
-importFolder "[created timecode folder]"`
-indexdb "[database GUID]"
Example:
powershell -nologo -noprofile -executionpolicy bypass -file .\IndexInfo.ps1 `
-dailyHost localhost `
-dailyACIPort 9000 `
-idxRootFolderPath ".\export" `
-importFolder "20201229T0825084082Z" `
-indexdb "MF-{17E2AF7E-D782-4573-84EF-94CAE7CD30A0}"
13. Start MFIndexingManager service from M-Files server. When started', the indexing starts from the beginning.
14. Optional. If you want to add more indexers to speed up the process go to the admin tool. Configurations > Advanced Vault Settings > Configuration > Search > Indexes > Index GUID > Additional Options. Change the value of Number of Parallel Indexers. Preferred maximum number is half of the cores in the M-Files server. Naturally, the amount also depends on the load: Each indexer uses about 11% of overall CPU power.
5. Re-index method 4: Full, non-destructive, parallel index
This method can also be used from the admin tool and it does not need playing with the scripts. On the other hand, it will need extra CPU power from the servers and extra space and engines from the IDOL side. The idea is to create a new, parallel index from the vault, while other indexes work as previously, until the new one is ready. Then you just change the active combined index into the new one. It should be noted that the new index will have its own indexers using ~11% of M-Files server CPU each and naturally they will double the disk usage and potentially rise engine amount from the IDOL side. Also, IDOL's compaction should be considered, which will again double the index size for the time of the process. For example, if our original index uses 100 GB, the new index will also use 100 GB- and the compaction will double it into 400 GB maximum.
1. Open the admin tool.
2. Select the vault for re-indexing.
3. Navigate into Configurations > Advanced Vault Settings > Configuration > Search > Indexes.
4. Click Add Index.
5. Add the needed settings.
NOTE 1: DO NOT COPY THE DATA FROM JSON OF THE OLD INDEX.
6. Save the settings. Indexing of the new, parallel index starts.
7. After the indexing has ended, change Active Combined Index from branch Configurations > Advanced Vault Settings > Configuration > Search into the name of the new index. Now the searches are done from the new index.
