Overview
This article describes one method to handle a large migration indexing during production use. The search and indexing of the new, manually inserted, and edited content must be available for searching long before the migration indexing is finished.
That is challenging, because we only have one indexing queue. In addition, we cannot say to the "old" index that do not index this and that specifically. In other words, when things change in the database, the system will start to index.
By using parallel indexing and modification of indexing timestamp we can achieve both goals: fast indexing and ultimately, all migrated material in the index. The downside of this method is that it utilizes more resources from the M-Files server and IDOL cluster.
Things to consider before migration
It should be noted that the parallel index will use the same number of indexers (CPU and memory) as the old one. Also, the space needed for the index will be doubled until you remove the old index. Last, in addition to needed space, IDOL cluster will be more burdened because of the parallel index. It is probably a good idea to check the current CPU and disk load before the operation and make an educated guess, whether the cluster can handle the indexing of the new index while the old one is still running.
The procedure
1. Get the current, largest timestamp from M-Files database.
SELECT MAX(VERSIONFORMDI) as max_items FROM OBJECTTYPEITEM;
2. Start the migration/changes.
3. Let the new material/translations come to the vault. Each new and updateable object will get a new timestamp to the M-Files database.
4. Again, get the largest timestamp from M-Files database. Now you should have two timestamps: One from the time before the migration and one after.
5. Stop the MFIndexingManager service.
6. Change the indexing timestamp in IDOL (i.e. the timestamp that the system thinks is the latest already indexed timestamp). To do that, follow Chapter 4 from the following article: Instead of changing the timestamp value into 0, change it into the wanted value.
7. Start the Indexing Manager Service. Now the changes that the users make will appear to the index immediately, but we do not have updated information from the migration in the index.
Alternatively, to steps below, there is a possibility to use previous steps to index the new data during off-working hours. However, that is not recommended, because there is always a chance that some data is not indexed e.g. due to human error with the timestamps. That might lead to the incomplete index that has to be rebuilt from scratch to make sure it is complete. We do not have means to validate the index against real world.
8. Create a new parallel index and wait for it to be finished.
9. Change the new index to Active Combined Index. Now searches use the new index.
10. After verifying, that searches are OK, disable and remove the old index.
