Overview
IDOL index can be expanded after the initial deployment by adding extra engines to the cluster. Currently, up to 16 engines is recommended to be used in a single cluster meaning a bit under 20 million documents per index. Adding the new engines is quite straightforward operation where the content of the index will be divided into the new engines.This document describes how to add new engines into existing cluster and how to divide the index for the new engines.
NOTE 1! This document is valid for IDOL version 12.x and beyond.
NOTE 2! Using this document requires basic understanding of IDOL components and terminology.
Keywords: IDOL, cluster, engine, addition
1. Prerequisites
- Existing IDOL 12.x cluster
- Setup tools for IDOL 12. Because the procedure uses some installation scripts and the installation instructions, the setup tools for IDOL 12 should be available.
- MFIDOLConsole. That is used for verifying the functionality of addition and for investigating problem situations. It can be found also from Setup tools for IDOL 12.
- Decide, which method, DREREDISTRIBUTE or export/import you can/want to use for dividing the index content into the new engines. The first one is much simpler, only one command, but it does not work with the cluster including more than one server. In addition, DREDISTRIBUTE does not work with the new distribution settings in dih.cfg:
DistributeByReference=FALSE
UseConsistentHashing=FALSE
Lastly, it will not create any backup files.
If you choose the latter, take the export now by referring to the article IDOL - Export and Import.
2. Steps to add new engines into the existing cluster
1. In M-Files server, stop MFIndexingManager service.
2. During the initial IDOL installation you have used IDOL12_setup_tool.xlsm to plan the cluster and to generate the installation scripts. You have also saved that file to have the current configuration for future usage. Now is that time. Open the file.
3. Select the last content engine line (whole line) and copy it.
4. Insert the copied line into the last free line after the content engine.
Result: The line is copied under the last content engine and the conflict appears due to the same port numbers.
5. Repeat the two previous steps until you have the amount of wanted new engines in the spreadsheet.
6. In the new engines, change the port numbers and select Solo install: Yes.
7. If you have previously added new engines with the solo install, change any previous Solo: yes to empty.
8. Generate scripts.
Result: A new installation scripts have been generated to the installation folder. The notable one is that has SOLO_INSTALL on it.
Example: 127.0.0.1.SOLO_INSTALL.bat
9. Open the command prompt with administrative rights and run the solo installation script.
Result: If the script is successful, it will tell few manual steps you have to do to make the cluster aware of the new engine.
10. Add new engines to the DIH's and DAH's configuration files as instructed in the previous step's Result.
Example: We have added two extra engines using port ranges 13000 and 14000. In following lines, changes are in orange color and added lines are in blue color. The usual location of dih.cfg and dah.cfg is C:\IDOL12\bin\[installation name]\dih and C:\IDOL12\bin\[installation name]\dah.
dih.cfg
[DIHEngines]
Number=4
[DIHEngine0]
Host = 10.1.8.122
Port = 11000
[DIHEngine1]
Host = 10.1.8.122
Port = 12000
[DIHEngine2]
Host = 10.1.8.122
Port = 13000
[DIHEngine3]
Host = 10.1.8.122
Port = 14000
dah.cfg
///[Server]
DistributedEngines=5
……
[DistributedEngine0]
Host = 10.1.8.122
Port = 9000
[DistributedEngine1]
Host = 10.1.8.122
Port = 11000
[DistributedEngine2]
Host = 10.1.8.122
Port = 12000
[DistributedEngine3]
Host = 10.1.8.122
Port = 13000
[DistributedEngine4]
Host = 10.1.8.122
Port = 14000
11. Restart DIH and DAH services.
12. Now it is time to divide the contents of the index into the new engines. In the Prerequisites section you have selected either a command DREREDISTRIBUTE or export/import method.
If you are using the simpler DREREDISTRIBUTE command, just run following URL in the Frontend server's browser and the cluster will automatically divide its contents into the new engines:
http://127.0.0.1:[DIH index port]/DREREDISTRIBUTE
Example: http://127.0.0.1:19001/DREREDISTRIBUTE
If you go with the export/import method use the export you already took in the Prerequisites section and import it by referring to the article IDOL - Export and Import.
