The patch installed suggests that it is necessary to rebuild the TrueSight Capacity Optimization (TSCO) Search index after installation. How can I manually rebuild the TSCO Search Index?
The following video shows how to rebuild the search index in TSCO:
Simple index rebuild
An immediate rebuild can be initiated via the TrueSight Capacity Optimization (TSCO) console.
The steps to manually rebuild the search index on TSCO are:
TSCO 11.5.01 and later
Access the TSCO console UI
Navigate to Administration > System > Status
Find the 'Data Hub' component and click on it
In the 'Data Hub' component list find the 'API Providers - Search service' and click on it
In the 'API Providers - Search service' screen click on the buttons 'Rebuild main index' and 'Rebuild tag index'
TSCO 11.3.01 and earlier
Access the Console
Navigate to Administration > COMPONENTS > Backend Services
Select API Providers - Search service in the table
Click on the buttons Rebuild main index and Rebuild tag index
You can find more details in official product documentation.
Advanced index rebuild, when the first procedure gives back errors
Resolution for the following error when rebuilding main index:
To resolve BCO_DH_ERROR11: "failed to acquire random test lock" error (all the commands need to me run from the TSCO installation directory, default /opt/bmc/BCO):
Stop all TSCO components that lock the TSCO index files. To do this, you have to run the command ./cpit stop on all the Application Servers.
Backup and then remove the existing search index files
cd repository
tar -zcvf search_index_bad.tar.gz search_index
cd search_index
rm -rf idx_cpit
rm -rf idx_tag
That will create a backup of your current search_index directory and then the rm -rf * will remove the current search_index contents.
WARNING: Be cautious when running an rm -rf since if the command is executed from the wrong directory it could result critical files being removed from the BCO environment which must be restored from backup to recover the environment.
Restart the TSCO components that were stopped in step 1 with the command ./cpit start
The Search Index rebuild will start automatically and it can take a few minutes to complete.
You can check the search index service logs under:
Administration -> Components -> Backend Services -> API Providers - Search service than click on Show Log
Administration -> ETL & System Tasks -> System Tasks -> Search index Updater than click on Show Log
You can also check the Search Index Updater (system task ID 34) logs: this task will search for the present of the index, once its not found as it was deleted, it will rebuild it, you will see the following lines in this log: [taskid=34]-Index is no present. Built it. [taskid=34]-Building CPIT_INDEX index...
Q: When the Search index is cleaned up this way, what task within TSCO triggers the rebuild?
The rebuild of the Search index after it has been removed it triggered by the execution of the Search Index Updater system task. This means that if the TSCO Primary Scheduler isn't restarted after the index is removed the rebuild won't be triggered. The TSCO Datahub will wait for a few minutes for the rebuild to be triggered but if it isn't triggered within a few minutes it will start without the index files being available (generating an error on startup).
If problem persists...
If these steps doesn't solve the problem, check also that the nfslock demon is running on all TSCO Application Servers.
How to debug file locking issues via command line (if search index isn't created, and if search service isn't deployed).
Check if these flock commands are useful to check the lock works OUTSIDE of the application. Execute them as cpit user on repository and see if they end within 1 second, fail, or remain stuck.
If they fail or remain hung there is an IT issue here, with the file locking on the repository volume (because it is mounted from an external NAS, for example) We recommend to work with IT to check what is wrong here (i.e. nfslock daemon stuck for example)
How to move the search index in a different folder outside repository without file locking issue (workaround)
This can be done ONLY if web, datahub (servicecontainer) running the search index are on the SAME HOST.
1) Create the target folder as cpit user via command line on the AS mkdir /opt/bmc/BCO/search_index
2) Access web console as admin user - navigate to administration- > system -> global configuration - press the edit button to edit properties - go in manual edit mode pressing the link "You can manually edit system properties from this page." at the bottom - find and edit the property search.repository.dir from ${REPO_DIR}/search_index/
to
${BASE_DIR}/search_index/
3) Via command line restart components to apply changes . ./cpit restart
4) Logon to the web console as admin user and launch the "search index updater". monitor via command line of the new search_index folder gets populated (the task might take several minutes to complete at the first run).