This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Go through below check points for configuration checks.

1] Restart the services in the following order:

1. RSSO
2. AR System Server
3. Mid-Tier
4. Smart IT
   

2] Validate the AR server pointing to Smart IT has RSSO integration installed or not?
First check if AR server connection with Smart IT is correct. In case there are multiple AR server lets say AR1, AR2, AR3.
And RSSO-AR integration patch is applied on AR2 only then make sure AR2 is set on below form and table.
Open form : SMT:Administration Console in mid-tier.
Select third entry in Application (MyIT-ITSM).
Check if correct AR server is set in Data Store.
If not then select AR2.
If AR2 is not listed then go to Persistent Data Stores option.
Add new record for hostname = AR2 and save.
Go back to Application and select AR2 in Data Store for MyIT-ITSM and save.

Note : Kindly access the form : SMT:Administration Console directly modifying the URL/Hyperlink and do not use old form : BMC MyIT Administration Console from Application Fly-out for configuration changes.

Similarly, check the configuration in SmartIT_System.CONFIGURATION_PARAMETER table in Smart IT DB for setting: connect.arsystem.hostName.
Update the server to AR2 if different AR server is set.

In case of load balancer, we need to check that all AR server in load balancer has RSSO-AR integration patch and ar load balancer is set as per above configuration.

3] Validate below configuration from AR system Server.
Check below configuration from AR end for below message in smart it debug logs.
User login_id - AServerUser is Null in UserSessionInfo

ARSystemServerInstallDir/conf/ar.cfg or ar.conf
Operating-Mode = 0
If it is set to 1 then set it to 0 and restart AR services and then Smart IT services.

4] RSSO configuration checks.
Configuration 1:
RSSO Admin Console -> Realm -> Authentication tab -> Tenant = tenant.com <dwp advanced catalog requires users to login like hannah_admin@calbroservice.com>

User ID Transformation= ToUpperCase

SmartITTomcat\External-conf\sso-sdk.properties = ignore-tenant=true

<SmartITInstallDir>\Smart_IT\Smart_IT\smartit\WEB-INF\classes\sso-sdk.properties = ignore-tenant=true

DWPTomcat\External-conf\sso-sdk.properties = ignore-tenant=true

SmartITTomcat/External-conf/rsso-agent.properties and <SmartITInstallDir>\Smart_IT\Smart_IT\smartit\WEB-INF\classes\rsso-agent.properties = The value of 'agent-id' property in the rsso-agent.properties file must be a unique identifier, and must be same on all nodes in SmartIT/BMC Digital Workplace cluster. It is recommended to set its value to a simple identifier instead of a HTTP URL. For example,

agent-id=smartit_agent.

SAML_AUTHENTICATION = 1

Configuration 2:

RSSO Admin Console -> Realm -> Authentication tab -> Tenant = blank

User ID Transformation= RemoveEmailDomain

SmartITTomcat\External-conf\sso-sdk.properties = ignore-tenant=true

DWPTomcat\External-conf\sso-sdk.properties = ignore-tenant=true

SmartITTomcat/External-conf/rsso-agent.properties and <SmartITInstallDir>\Smart_IT\Smart_IT\smartit\WEB-INF\classes\rsso-agent.properties = The value of 'agent-id' property in the rsso-agent.properties file must be a unique identifier, and must be same on all nodes in SmartIT/BMC Digital Workplace cluster. It is recommended to set its value to a simple identifier instead of a HTTP URL. For example,

agent-id=smartit_agent.

SAML_AUTHENTICATION = 1

5] If SSL is involved (rsso/smart it urls are ssl enabled -https)
https://docs.bmc.com/docs/rsso91/manually-integrating-remedy-sso-with-remedy-applications-799091345.html#ManuallyintegratingRemedySSOwithRemedyapplications-CAR
Also verify rsso-agent.properties file configurations, if  ssl based urls are being used then please make sure following parameter should be set to true in rsso.cfg as well as rsso-agent.properties files.

ARSystemServerInstallDir/conf/rsso.cfg
com.bmc.rsso.tls.disable.checks= true

Also add this parameter in rsso-agent.properties file on Smart IT servers tomcat.
SmartITTomcat/External-conf/rsso-agent.properties
com.bmc.rsso.tls.disable.checks= true

6] Check rsso related libraries on Smart IT Tomcat folder
https://docs.bmc.com/docs/smartit2002/integrating-bmc-remedy-sso-with-smart-it-908202563.html
(Only for upgrade) If you want to get the new Remedy SSO features, clear the old Remedy SSO jar files by stopping Smart IT, and copy the following jar files from the Remedy SSO machine Installer Path \Disk1\files\lib, and paste the files at SmartITTomcat/External-conf/lib:
caffeine-x.x.x.jar
gson-x.x.x.jar
jackson-annotations-x.x.x.jar
jackson-core-x.x.x.jar
jackson-databind-x.x.x.x.jar
jjwt-api-x.xx.x.jar
jjwt-impl-x.xx.x.jar
jjwt-jackson-x.xx.x.jar
json-20180813.jar
org.apache.oltu.oauth2.client-x.x.x.jar
slf4j-api-x.x.xx.jar

(Only for upgrade) Copy the following files from the Disk1/files/rsso-agent folder to SmartITTomcat\External-conf\lib:
rsso-client-impl.jar
rsso-sdk-atsso.jar
rsso-agent-all.jar

7] Check arjavaplugin.log from AR Server and check if you find below errors related to ARSYS.AREA.RSSO plugin.
Log location: ARSystemServerInstallDir\ARSystem\Arserver\Db\arjavaplugin.log

<PLUGINSVR> <TrID: nZ9PmtHGS4ifeglHEmAVdw:0000001> <TNAME: Thread-96                > <ERROR> <ARPluginContext                                   > <                              ARPluginContext.java:217       >   <ARSYS.AREA.RSSO>Exception on initialization: Could not register consumer 'ar_plugin' at server 'sub-dmn:http://rssoserver/rsso'. Make sure you are using server >= 18.08.00. Cause: Could not register consumer 'ar_plugin' at server 'sub-dmn:http://rssoserver/rsso'. Make sure you are using server >= 18.08.00. Stacktrace: [com.bmc.rsso.sdk.SSOServiceLib.getService(SSOServiceLib.java:83), com.bmc.rsso.plugin.area.RSSOPlugin.getSsoService(RSSOPlugin.java:86), com.bmc.rsso.plugin.area.RSSOPlugin.initialize(RSSOPlugin.java:165), com.bmc.arsys.pluginsvr.plugins.ARPluginLoaderRouter.initializePlugin(ARPluginLoaderRouter.java:877), com.bmc.arsys.pluginsvr.plugins.ARPluginLoaderRouter.loadPlugin(ARPluginLoaderRouter.java:788), com.bmc.arsys.pluginsvr.plugins.ARPluginLoaderRouter.instantiateAllPlugins(ARPluginLoaderRouter.java:768), com.bmc.arsys.pluginsvr.binding.ARPluginDynamicInstantiateHandler$DynamiclyInstantiateOneJavaPlugin.run(ARPluginDynamicInstantiateHandler.java:75), java.lang.Thread.run(Unknown Source)]
<PLUGINSVR> <TrID: nZ9PmtHGS4ifeglHEmAVdw:0000001> <TNAME: Thread-96                > <ERROR> <ARPluginLoaderRouter                              > <                         ARPluginLoaderRouter.java:835       >Thread 249 failed to create an instance of ARSYS.AREA.RSSO

The fix is available for RSSO/Smart IT 19.08 and 20.02.
Download the one applies to your environment and test.
OR apply latest RSSO hot fix for specific version.
Defect for 1908: DRSMX-26174
Defect for 20.02: DRSMX-50502

Steps to implement the jar file:
Stop the AR System server.
Backup the ARSystemServerInstallDir/pluginsvr/rsso-area-plugin-all.jar file
Replace it with rsso-area-plugin-all.jar file from hotfix.
Restart the AR System server

=============================================
NOTE***:
At smartit startup, it (smartit) copy file from classes (C:\Program Files\BMC Software\Smart_IT\Smart_IT\smartit\WEB-INF\classes) to <TomcatInstallDir>\external-conf\ if not exists in  external-conf.
It can  be remove file after copying in such cases.
The rsso and sso file from the same location (C:\Program Files\BMC Software\Smart_IT\Smart_IT\smartit\WEB-INF\classes) can be renamed OR removed, so that SmartIT can read from <TomcatInstallDir>\external conf \ location.

After comparing working and non-working logs of Smart IT.

Working Logs:
DEBUG   | c.b.b.m.f.h.AbstractLoginHandler | isAuthenticationRequest(): path=/users/sessions, result=true, method=POST
DEBUG   | c.b.b.m.f.h.SAMLLoginRequestHandler | User: USERNAME, Transform unknown
 WARN    | c.b.b.m.f.h.SAMLLoginRequestHandler | Unknown transform unknown
DEBUG   | c.b.b.m.f.h.SAMLLoginRequestHandler | getAuthenticatedUser - user: USERNAME, token: _29b2f2a1-dcfd-400c-bf85-05be3e119872
DEBUG   | c.b.b.m.f.AuthenticationFilter | Auth handler: 'com.bmc.bsm.myit.filter.handlers.SAMLLoginRequestHandler', requestState: 'PASS'
c.b.b.m.f.AuthenticationFilter | After invocation of all authentication handlers request status is: 'PASS'
 INFO    | c.b.b.m.f.r.UserPrincipalRequestWrapper | principal.getName():
INFO    | c.b.b.m.f.r.UserPrincipalRequestWrapper | UserPrincipalRequestWrapper.getUser: USERNAME
 INFO    | c.b.b.m.f.r.UserPrincipalRequestWrapper | UserPrincipalRequestWrapper.getUser: USERNAME
DEBUG   | c.b.b.m.s.s.DefaultSecurityService | Creating authentication...
INFO    | c.b.b.m.service.LoginServiceImpl | LoginID2: USERNAME, SSO Token2: _29b2f2a1-dcfd-400c-bf85-05be3e119872
DEBUG   | c.b.b.m.service.SecurityService | loginId=USERNAME, authentication=_29b2f2a1-dcfd-400c-bf85-05be3e119872, tenantId=000000000000001, systemApplicationName=MyIT, userApplicationName=MyIT-ITSM, applicationVersion=9.1.08.000,apiVersion=9010800, clientLocale=en_US, deviceToken=dummyToken, osVersion=Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; Tablet PC 2.0; wbx 1.0.0; rv:11.0) like Gecko, deviceModel=Web Client, verify=false
DEBUG   | c.b.b.m.service.SecurityService | Entering login(USERNAME, *****), tenantId 000000000000001
 DEBUG   | c.b.b.m.service.SecurityService | Exiting login()

In non-working logs,
 DEBUG   | c.b.b.m.f.h.AbstractLoginHandler | isAuthenticationRequest(): path=/users/sessions, result=true, method=POST
DEBUG   | c.b.b.m.f.h.SAMLLoginRequestHandler | User: USERNAME@tenant.com, Transform unknown
WARN    | c.b.b.m.f.h.SAMLLoginRequestHandler | Unknown transform unknown
DEBUG   | c.b.b.m.f.h.SAMLLoginRequestHandler | getAuthenticatedUser - user: USERNAME@tenant.com, token: _4d99dabf-d079-4ef4-a4dc-6b6c45285df4
DEBUG   | c.b.b.m.f.AuthenticationFilter | Request is authenticated
INFO    | c.b.b.m.f.r.UserPrincipalRequestWrapper | UserPrincipalRequestWrapper.getUser: USERNAME@tenant.com
INFO    | c.b.b.m.f.r.UserPrincipalRequestWrapper | principal.getName():
INFO    | c.b.b.m.service.LoginServiceImpl | Start SSO Login
INFO    | c.b.b.m.service.LoginServiceImpl | LoginID2: USERNAME@tenant.com, SSO Token2: _4d99dabf-d079-4ef4-a4dc-6b6c45285df4
DEBUG   | c.b.b.m.service.SecurityService | loginId=USERNAME@tenant.com, authentication=_4d99dabf-d079-4ef4-a4dc-6b6c45285df4, tenantId=000000000000001, systemApplicationName=MyIT, userApplicationName=MyIT-ITSM, applicationVersion=9.1.08.000,apiVersion=9010800, clientLocale=en_US, deviceToken=dummyToken, osVersion=Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36, deviceModel=Web Client, verify=false
18/Oct/2019:14:22:37 -0400 | https-jsse-nio-443-exec-5      | RID: 40 | DEBUG   | c.b.b.m.service.SecurityService | Entering login(USERNAME@tenant.com, *****), tenantId 000000000000001
18/Oct/2019:14:22:37 -0400 | https-jsse-nio-443-exec-5      | RID: 40 | ERROR   | c.b.b.m.r.p.MobilityExceptionMapperProvider |
com.bmc.bsm.mobile.errorhandling.MobilityException: {"error":"MOBILITY_ERROR_LOGIN","errorCode":1006,"defaultMessage":"Login Failed: An authentication error occurred in the data server.","additionalMessage":"Authentication failed","detailMessage":"ERROR (623): Authentication failed; Incorrect username or password","ARConnectionProblem":false}
            at com.bmc.bsm.mobile.utils.ErrorResourcesHandler.checkARLoginError(ErrorResourcesHandler.java:91)
            at com.bmc.bsm.mobile.vo.profile.MobilityARUser.verifyUser(MobilityARUser.java:136)
Caused by: com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Incorrect username or password
            at com.bmc.bsm.mobile.utils.ErrorResourcesHandler.checkARLoginError(ErrorResourcesHandler.java:87)
            ... 110 common frames omitted
DEBUG   | c.b.b.myit.filter.ExpiresFilter | Request '/smartit/rest/users/sessions' with response status '500' content-type 'application/json', set expiration date Fri Oct 18 14:22:37 EDT 2019

here,
USERNAME = Name of the User who is attempting to login.

NOTE: Also, make sure you have RSSO is working with mid-tier and verify below checks as well.
000331616 - Remedy Single Sign On - Troubleshooting Steps for ARERR 623 Authentication Failed with Remedy
https://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA33n000000Y9PdCAK&type=Solution


--------------------------
Additionally in arjavaplugin log check if there is the presence of  following
<ARSYS.AREA.RSSO>Exception on initialization: Could not register consumer 'ar_plugin' at server 'https://<domain>:443/rsso'. Make sure you are using server >= 18.08.00. Cause: Could not register consumer 'ar_plugin' at server 'https://<domain>:443/rsso'. Make sure you are using server >= 18.08.00. Stacktrace:

in one case, rsso.cfg had an old domain reference.
---------------------------



See also: BEST FAQ on Remedy for Smart IT Connectivity Issues