This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Full Text Search (FTS) – Deployment and Configuration Best Practices

This document is a whitepaper on FTS Deployment and Configuration Best Practices.
Introduction
The Full Text Search (FTS) is a feature of the ARServer that enables faster searches when the searches are performed on FTS-enabled fields of a given AR Form.
In contrast to the standard character-based search function available in a database, FTS is a word-based search thus providing faster results when searching for words or phrases.
FTS offers several benefits over database such as fast, indexed searching of CLOB fields, searching within attached files, word-stemming, weighting search results, and more.
FTS is also an integral part of ITSM and Smart IT use cases. Thus, an optimally configured and properly sized FTS environment will result in an optimal ITSM and Smart IT performance.

This whitepaper shows how to optimally configure your FTS environment including FTS load distribution.



How-To Enable FTS
To include a field of any AR Form in Full-Text Searches, enable the field for FTS using Developer Studio.

          

When a field is enabled for FTS, any data added, removed, or updated to the field are automatically indexed by the Lucene engine, an open source embedded FTS product.



Overview of how an FTS server works
Another technical perspective:
The FTS service contains an indexing service (item 1) that writes and updates a set of index files (item 3); and a search service that reads the index files and uses the data to perform fast searches.
The ARServer together with the FTS searcher running in its own plug-in server is called the FTS Server (in the context of an ARServer group).

Events such as create, update, and delete within the AR System (e.g. ARServer group) are queued via the ft_pending table in the database.
The FTS Server spawns a set of indexer threads taking items from this queue to add, delete, or update the data within its index files accordingly.
Only the ARServers designated as FTS Servers service the FTS queue. However, any ARServer within an ARServer group adds to this queue when servicing an AR API call that alters data on a field marked for FTS.

The sequence of data flow is as follows:
- When a non-FTS Server in an ARServer group is servicing an AR API call needing to perform an FTS search, it sends a search request to the FTS Searcher plug-in.
- The FTS Searcher plug-in performs the search and returns the result to the caller ARServer.
- The ARServer caller packages the search results into the AR API response of the original AR API call.
NOTE: An FTS Server performs its own FTS searches and does not need to forward any FTS search requests.



Configuring FTS Service in an ARServer Group
When configuring the FTS service for an ARServer group, do the following:
-- Identify and Rank (in the AR System Server Group Operation Ranking form) any servers in the Server Group as FTS Servers (more than 1 can be FTS Servers for HA and workload distribution purposes) --

> Each FTS Servers will have its own local copy of the FTS index files.
> Each FTS Server provides FTS service for itself and other members in the ARServer group.
> The primary FTS server (Ranked #1) serves as the default Searcher server.
> When the primary FTS Server is down or unavailable, another will be chosen as the primary FTS Server based on the configuration in the ranking form.

You can designate any ARServer in an ARServer group as an FTS Server by ranking it in the AR System Server Group Operation Ranking form.
Each configured FTS server always services the FTS queue regardless of its ranking order.

IMPORTANT: A running FTS service can consume a large amount of resources (CPU, Memory, Disk Space); thus, FTS Servers are not recommended to be User-facing servers.

For more information, see Setting failover rankings for servers and operations.



How failover occurs in high-availability configuration
FTS requires that at least one of the servers in a server group act as a primary FTS indexing server.
To provide failover, you can have two or more FTS indexing servers in the server group where each server acts as an independent FTS Indexing server and maintains a local folder of the FTS Index in its Collection directory.
When an indexing action takes place, each FTS indexing server indexes it independently.

For example, if you create a form and then index a field on that form, each FTS indexing server indexes that field. Because you can have multiple FTS servers, each with its own current copy of the FTS data, you can fail over to a second server as needed.

An AR System server is designated as an FTS indexing server by having a ranking record for FTS in the AR System Server Group Operation Ranking form.
Each FTS server defined in the ranking form acts as an indexing server and can provide FTS search services to other servers in the server group.
Each defined FTS server will always index, regardless of its ranking order.

FTS servers can consume large amounts of resources (CPU, Memory, Disk Space) and are recommended to NOT be located on User-facing servers.
Servers in the server group that are not ranked for FTS are Searcher servers.
The Searcher servers are user facing or non-FTS back end servers that connect to one of the FTS indexing servers when they need to search the FTS data.

Note: The servers that are not designated as indexing server should not be listed in the AR System Server Group Operation Ranking form.

The following figure shows the FTS Servers in a high-availability architecture:

          BMC Remedy FTS Servers in a high-availability architecture

          


The default configuration for all non-FTS servers in an ARServer group is to connect to the Primary FTS server (ranked 1 in the AR System Server Group Operation Ranking form).
This is specified in the Server-Plugin-Alias entry in the Centralized Configuration and ar.cfg [or ar.conf] files of the AR Servers.

For example, consider a scenario where you have two FTS Servers that are ranked 1 and 2 in the AR System Server Group Operation Ranking form.
Then your Server-Plugin-Alias entry has the value of the server name that is ranked 1.
- If a request to this primary FTS Server errs, the request is then sent to the next FTS Server that is ranked 2 in the AR System Server Group Operation Ranking form.
- If the request sent to the FTS Server ranked 2 also errs, then an error is returned since no FTS Server is available.



Architecture for FTS setup
For some FTS deployment, it’s sufficient for all searches and indexing from the user facing servers be handled by one primary FTS Server with Rank 1. Therefore, the rank 2 FTS Server functions as HA and no searches are sent to this server unless the primary FTS Server is down or unavailable. This is the default behavior for FTS Servers in an ARServer Group.

To overcome moderate heavy loads on a single FTS Server, there are two possible configurations:

Configuring multiple FTS Searchers without a Load Balancer :
Each user-facing ARServer is configured to direct searches to a specific FTS Server.
An FTS Server can provide the service to one or more User Facing servers as shown in the diagram below.
In the case that the manually configured FTS Server is down or unavailable to accept a search request, the ARServer redirects the search to another AR Server starting with Rank 1,2,3 until one of the FTS Servers is available to handle the request.
           Configuring multiple FTS Searchers with a Load Balancer
A load balancer is configured between the User Facing AR Servers and the FTS Servers. In this case if any searcher becomes unavailable the load balancer will redirect the search query to another Searcher Plugin. If the FTS Server is busy performing a re-index and cannot accept the search request, the load balancer may still direct searches to it. In this case, the ARServer automatically redirects the request to FTS Servers starting with Rank 1,2,3 until one of the FTS Server is available to handle the request.
          
Note:  What to do if your indexes become out of sync.
If you have an Indexer server go down for some period of time, such as for performing routine maintenance, you may have a situation in which the ft_pending jobs for a single server are piling up. 
When you bring that server back up, it will take time to catch up with the queue. 
But because that server has become available, searches will still be performed, giving inconsistent results to end users.
It is best to prevent users from searching on this indexer until it is caught up.
As mentioned earlier, you can run the following SQL to  check how out of sync your indexers have become:
select count(*), indexservername from ft_pending group by indexservername;

Once you bring the server back up, immediately stop the fts searcher plugin server.
This can be done easily in v.18.05 and v.18.08 and v.9.1.04 where the Server Group Administration Console has been installed.
For 9.1.04, this is an option available from https://communities.bmc.com/docs/DOC-102081

From the Server Group Administration Console:
1- Open the "Manage Processes" page.
2- Sort by Process Name and look for the BMC:FtsJavaPluginServer entry.
3- Identify the exact line for the server that is out of Sync.
4- Highlight that line and choose an Operation of "Stop". 
5- Click on the Execute button. 
    This will stop the plugin server and searches will not be performed on this Indexer server.
6- Periodically run the SQL until your indexers are in-sync.
7- Then highlight the stopped BMC:FtsJavaPluginServer and choose Operation 'Start' and execute.

Searches will now be directed to this indexer server.



Load Test Results
In BMC’s performance lab we have configured a SmartIT cluster Environment to test FTS.
We have:
> 6 user facing servers
> 2 FTS indexers with Rank 1 and 2 as shown in below diagram.
> Load Tests were conducted with a Load Balancer Configuration in place.

All these tests were carried out for search scenarios.
          

Test #1 – A test was performed in which we had search requests distributed across two admin indexers using LB.
Following are the details of the tests carried out and the conclusion.
            Conclusions :
            1- The Response time increases marginally as the threshold increases.
            2- The Transactions Per Second is negatively impacted as the threshold increases.
                Search’s best performance is observed with 2k Threshold.
                With the increase of threshold, the total number of search transactions completed per second reduces along with an increasing trend in response times.
            3- There is significant difference in throughput viz. approximately additional 32k transactions per hour for 2k threshold compared to 10k threshold, and approximately additional 25k transactions per hour compared to 4k threshold.
 

Test #2 - A comparison study on searches for Response time(s) and transactions per second was done with 1 indexer Server and 2 Indexer Servers.   


Tests for Indexing
Tests details:
            - FTS threads: 3.
            - Indexer Node’s CPUs: 4.      
 
Test # 1: Time taken for an Incident to get indexed: Test # 2: Time taken for the records to be handled from ft_pending.  
Test # 3: Measured Time taken for Global Re-index.   


FTS plug-in configuration
Documentation:
v.9.1.x: https://docs.bmc.com/docs/ars91/en/fts-plug-in-configuration-609072952.html
v.18.05 (9.1.05): https://docs.bmc.com/docs/ars1805/fts-plug-in-configuration-804712974.html
v.18.08 (9.1.06): https://docs.bmc.com/docs/ars1808/fts-plug-in-configuration-820496274.html

The FTS plug-in is used to search Full Text Search indexes.
The FTS plug-in provides row-level and field-level security for indexed data during searches.
This plug-in supports the multiform search used by Global Search and in applications such as BMC Remedy Knowledge Management (RKM).
The multiform search uses a vendor form that displays an interface through which an application customizes the search to suit its requirements.
The search can contain a request to return a specific list of fields, how the filters are returned, or terms that may or may not be available in the document, and so on. It also permits you to specify the forms to search.
You can tailor the search to target just the indexed data rather than all data.

Plug-in type: 
FTS is a Java-based Filter API plug-in.

AR System server connectivity: 
The FTS plug-in interacts with the AR System server through the BMC Remedy AR System Java API.
The plug-in (ftspluginVerNum.jar — VerNum represents the release version number) is installed in the <ARInstallationFolder>\pluginsvr\fts folder.

Configuration information
The FTS plug-in is installed during the AR System installation and is hosted with the standard AR Java plug-in server.
Each instance of the AR Java plug-in server is permitted to run only one instance of the FTS plug-in.
By default, the FTS plug-in runs on port 9977. If you want to change the plug-in port value, the default port number value must be changed in both ar.cfg [or ar.conf] and pluginsvr_config.xml files.
You can also use Centralized Configurations generic UI.
In a single-server installation, ensure that the collection directory is on the same computer as the server. In a server group, ensure that all the FTS plug-in instances always point to the local collection directory. Configuring full text search

Link: Configuring full text search for a server group
The configuration information of the FTS plug-in is available in the <ARInstallationFolder>\pluginsvr\pluginsvr_config.xml file that includes the plug-in details in XML code.
The ar.cfg/ar.conf file includes the Server-Plugin-Alias setting that points to the correct plug-in server alias:
Server-Plugin-Alias: ARSYS.ARF.FTS ARSYS.ARF.FTS myServer:9977
The FTS plug-in is launched from the armonitor.cfg [or armonitor.conf] file located in the <ARInstallationFolder>\conf folder.
 
More Info:
https://docs.bmc.com/docs/display/public/ars91/Configuring+full+text+search

Configuration Consoles
Most FTS Configuration can be performed from the FTS Configuration page on each individual server.


          

This will automatically set the appropriate configurations in the ar.cfg/conf, armonitor.cfg/conf, and pluginsvr_config.xml
In 9.1.04 and later versions, you have the option of configuration all the servers in your group from a single Server Group Administration Console. 

         


Best Practice Configuration
The following section details the FTS Best Practice Settings based on environment size.
The configuration recommendations are specific to small, medium and large model defined as follows:
 

Environment	BMC Remedy ITSM (concurrent users)	BMC Remedy Smart Reporting (concurrent users)	Number of managed devices
Small	800	50	2,000 or fewer
Medium	2,000	100	10,000 or fewer
Large	5,000	250	30,000 or fewer

Configuration settings
 

Component	Small	Medium	Large
Number of Users	800	2,000	5,000
Physical RAM	24 GB	32 GB	64 GB*
ARServer Heap Size	10 GB	14 GB	32 GB*
Plugin Server Heap Size1	4 GB	6 GB	10 GB
Number of Core Threads2	10	12	16
FTS Queue (FTS Indexer servers)3	Private-RPC-Socket: 390602 3 3	Private-RPC-Socket: 390602 5 5	Private-RPC-Socket: 390602 7 7

*Recommend HA setup.

Note:

GC settings
BMC recommends using these GC parameters (same as ARServer) with Plugin Server for FTS.
.................................................
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+DisableExplicitGC
-XX:NewRatio=2
-XX:+UseCompressedOops
.................................................
 
For Example using the Small Model (armonitor.cfg) on FTS Indexer Server:
........................................
"C:\Program Files\Java\jdk1.8.0_73\jre\bin\java" -Xmx4096m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:NewRatio=2 -XX:+UseCompressedOops
-Dcom.sun.management.jmxremote.port=8004 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-classpath "C:\Program Files\BMC Software\ARSystem\pluginsvr\fts\secondary;C:\Program Files\BMC Software\ARSystem\pluginsvr\fts\core;C:\Program Files\BMC Software\ARSystem\pluginsvr;
C:\Program Files\BMC Software\ARSystem\pluginsvr\arpluginsvr91_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x { FTS Indexer AR_Server_Name} -i "C:\Program Files\BMC Software\ARSystem" -m
........................................




Troubleshooting
https://docs.bmc.com/docs/ars1805/troubleshooting-fts-plug-in-issues-805339782.html




FTS Monitoring
1- Check AR Server Ranking form (AR+System+Server+Group+Operation+Ranking).
    Ensure that the Full Text Index is set to Rank=1 for only the Primary Index server.
    Servers which are not serving Full Text Index should have Rank as Empty/Null or the Ranking entry removed.

2- In AR Server console under Server Information's ->FTS tab "Disable Full text searching"  and “Disable FTS  Indexer” checkbox should not be checked.

          



3- Under FTS Configuration check that FTS Agent checkbox is enabled.
          


4- Under Server Configuration -> “Server Group Indexer” radio button is selected for “Indexer Server” and “Server Group-Searcher” is selected for a Searcher server.

            


5- Monitor ft_pending table.
    Ensure that the entries are getting created in the ft_pending table for any FTS Operations.
    select count(*) from ft_pending;

    If this shows that the count is increasing and decreasing then it confirms that your indexing operations are being processed.
    - Ensure that the ServerName (user facing server) and IndexServerName (FTS Indexing Server ) are appropriate to the Environment and there are no other extraneous entries in the ft_pending table.
    - Any entries in the IndexServerName field other than the Indexing Server name will not be handled by ft_pending.
      Those entries can be deleted.
    - If the ft_pending table is properly handled by FTS Indexer, the updateTime will be updated with -1 if its data operation and it will be a numeric count if it’s a meta-data operation like global reindex on the schema.
      See Knowledge article 000152517 for more information.
    - You can use the Server Group Administration Console to Monitor indexing activity.


6- Monitoring using JVisualVM: FTS Thread Monitoring.
    a) Under threads monitoring in JVisualVM, check the pool-FTS-threads count.
        These threads are used to handle data operations from FTS like CREATE_ENTRY/SET_ENTRY /   INDEX_ENTRY ...etc.

    b) There will be n+1 threads where n is the number of threads set in ar.conf for Private-RPC-Socket: 390602.
        If Private-RPC-Socket: 390602 is set to 3 3, you will observe 4 threads under pool-FTS-threads.
 
         
 
    c) There are also another 4 threads for FTS (pool-FTS-SERIAL-threads).
        These threads are visible or started only when metadata operations like GLOBAL_REINDEX, SCHEMA_REINDEX , FIELD_REINDEX ..etc are triggered.

    d) There will be n+1 threads where n is the number of threads set in ar.conf for Private-RPC-Socket: 390602.
        If Private-RPC-Socket: 390602 is set to 3 3, you will observe 4 threads under pool-FTS-SERIAL-threads.
         
         



7- Check arftindx.log for information regarding index operation handling.