Symptoms:

1) TrueSight Server Automation (TSSA) Jobs run successfully with a low job parallelism setting (e.g. <50) but throw the following errors intermittently when the parallelism is increased:

JOB Run Log:

Error      05/14/2015 09:48:45        Network Shell can be used for local access
Error      05/14/2015 09:48:45        Error in Initializing RBAC User and Role (SSO Proxy)
Error      05/14/2015 09:48:45        SSL_connect
Error      05/14/2015 09:48:45        SSL_connect
Info        05/14/2015 09:49:50        Exit Code 1
Error      05/14/2015 09:49:51        Error running master script on server targethostname
Error      05/14/2015 09:49:51        Unable to create destination directory /install/SUS
Error      05/14/2015 09:49:51        //targethostname/c/: Connection timed out
Error      05/14/2015 09:49:51        SSL_connect
Error      05/14/2015 09:49:51        SSL_connect

2) The environment is using NSH Proxy Servers through which all NSH Traffic is routed.

3) The number of failed servers varies between runs as does the name of the failed targets so the issue is not an agent-side issue.

3) The SSL_connect errors are accompanied by a "Connection timed out" error in the job run log.

4) The appserver log contains the following, corresponding stack trace:

[14 May 2015 09:49:20,617] [Nsh-Proxy-Thread-555] [WARN] [Anonymous:Anonymous:10.200.29.74] [BLSSOPROXY] javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
com.bladelogic.sso.engine.ClientDisconnectedException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:123)
at com.bladelogic.sso.engine.StreamMessaging.receiveToken(StreamMessaging.java:203)
at com.bladelogic.om.infra.mfw.net.BlSessionServerConnection.authenticate(BlSessionServerConnection.java:176)
at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.internalHandshake(BlSessionNshServerConnection.java:78)
at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.doHandshake(BlSessionNshServerConnection.java:50)
at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.setupClient(BlSessionNshProxyPair.java:104)
at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.init(BlSessionNshProxyPair.java:75)
at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:106)
at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:17)
at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:119)
... 9 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(Unknown Source)
... 16 more
[14 May 2015 09:49:20,618] [Nsh-Proxy-Thread-555] [INFO] [Anonymous:Anonymous:10.200.29.74] [BLSSOPROXY] failure establishing session with proxy service

The "failure establishing session with proxy service" line at the end further suggests that this is an issue communicating with the NSH Proxy Server, which happens to be on the same host as the Job Server in this case but may not be, rather than an issue communicating with the Target Server.

Note: Please see the TSSA RSCD Agent Connectivity Troubleshooting Guide for more details on troubleshooting this issue.
 

The likely cause here is either:

1) There are not enough NSH Proxy Server Threads and/or Contexts available on the NSH Proxy Server(s).
2) There are far too many NSH Proxy Server Threads and/or Contexts configured on the NSH Proxy Server(s).
3) The NSH Proxy Server is not replying to all the inbound NSH requests from the Job Servers in sufficient time. The default timeout for the target of NSH requests (either NSH Proxy Servers or Target Servers) is 30 seconds


Solution:
For #1 and #2 above, make sure the MaxNshProxyContexts, MaxNshProxyThreads and NshProxyMaxThreadIdleTime NSH Proxy Server blasadmin settings are set in-line with the following recommendations:

docs.bmc.com/docs/display/public/bsa86/Recommendations+for+Application+Servers+of+type+NSH_Proxy

e.g.
 

blasadmin -a set app MaxNshProxyContexts 300
blasadmin -a set app MaxNshProxyThreads 60
blasadmin -a set app NshProxyMaxThreadIdleTime 0


BMC recommends setting NshProxyMaxThreadIdleTime to 0.
250/50 are reasonable values to set MaxNshProxyContexts/MaxNshProxyThreads initially but this completely depends on the number of NSH Proxy Servers in the environment, the heapsize of the NSH Proxy Deployment, the number of parallel NSH users and NSH-related jobs etc. Contact BMC Support for assistance if unsure how to set these values for your environment.

For #3 above, increase the default timeout for outbound NSH requests from the job servers. The default is 30 seconds. This can be increased to 180 seconds by adding the "timeout=180" entry to the default line of the secure file on the Job Servers e.g.

default:port=4750:protocol=5:tls_mode=encryption_only:auth_profile=defaultProfile:appserver_protocol=ssoproxy:encryption=tls:timeout=180

docs.bmc.com/docs/display/public/bsa86/Options+for+the+secure+file