BEST PRACTICE:
Offload SSL through loadbalancer instead.
If this is the case, then you will need to install the LB SSL certificate on DWP/DWPC Keystore/CA Certs.
Check with Citrix if there are additional considerations for NetScaler, etc.
POSSIBLE ROOT CAUSES:
- Bad certificate: missing signature: root/intermediate certificate not present, DNS/hosts entries are missing: Please, make sure you have at least hostname, fqdn in the dns entries, if you have an alias you should have aliashostname and aliasfqdn.
- Expired certificate
- When using wildcard certificates confirm it has all the san entries, i.e.
- My company has the following domains: mycompany.dwp.com, mycompany.dwp.mx, mycompany.dwp.org, alter.mycompany.dwp.com.. dwp.mycompany.alternative.com,
- Then multiple wildcard have to be declared when issuing a wildcard certificate: *.dwp.com, *.dwp.mx, *.dwp.org, *.mycompany.dwp.com, *.mycompany.alternative.com
- If the DWP/DWPC Loadbalancer has a cert, make sure to import the same to DWP and DWP C Servers. That will also apply for RSSO, Midtier, SmartIT loadbalancers.
- Server A cannot resolve dns entries from Server B or/and viceversa: This can be managed vi DNS Server or by adding entries under server's hosts file
- SSL port blocked
- If you have a DWP Catalog Cluster, make sure all the DNS entries are present in the DWPCatalog.jks/Keystore. Best practice would be creating the keystore as follows:
keytool -genkey -alias DWPC -keyalg RSA -keystore DWPCatalog.jks -dname "cn=ServergroupName/LoadBalancerName" -storepass keystorepass -deststoretype JKS -ext san=dns:loadbalancerfqdn,dns:server1,dns:server1fqdn,dns:server2,dns:server2fqdn,dns:server3,dns:server3fqdn
- Loadbalancer redirection issue
- Non root user trying to start a process on port 80/443 which can only be used by root users. Non root users cannot run applications on port 80/443.
- Several steps from the below documentation were skipped. Make sure that the DWP Catalog .crt file is imported into the DWP Adv keystore, as mentioned here:
RESOLVING THE ISSUE:
- In order to fix the PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target error on the DWP Advanced - Enhanced Catalog Section, do the following
- NOTE: Had one case on which, even though the certs were right, VIP and LB balancer settings were off, and customer was still getting above errors when using DWP Catalog LB URL under DWP Admin Enhanced Catalog section. As a workaround, we had to use DWP C primary server URL instead.
- Get the cert from DWP Catalog by using Firefox
- If you have DWP Advanced Cluster, make sure all the server group nodes have the DWP Catalog Cert. Make sure you do the same on DWP, AR Server, SmartIT and RSSO servers.
- Import the certificate to the cacerts by using below commands
- openssl x509 -outform der -in catalogcert.pem -out certificate.der
- keytool -import -alias DWPCatalog -trustcacerts cacerts -file certificate.der
Note: It can be java cacerts or a custom one
- Restart Applications Tomcat / AR server service
- Clear browser cache
- Log into the DWP Admin Console.
- Go to the Enhanced Catalog section and update the DWP Catalog url with the right port and protocol.
- Do the same from the DWP Advanced LB URL and from each node individually to confirm this works.
- Do the same on the AR Server's SB:RemoteApprovalConfiguration and Centralized Configuration com.bmc.itsm.sbe
- Click on Save
- Confirm that you are not getting the error anymore
- If you get the error proceed with SSL Poke Troubleshooting section below
SSL POKE
- Download SSL poke and run it on DWP Advanced Server. SSL Poke is attached to this KA.
Consider you will have to repeat this from AR Server, RSSO, SmartIT Servers to DWP Catalog and viceversa.
- On a cmd/terminal Windows Run:
java SSLPoke DWPCatalogServerFQDN SSLPort
Example:
java SSLPoke dwpcatalogservername.com 8443
- If Firewall allows and you have imported a good cert to DWP Server, you should get a message like "Connection was successful"
- If it fails, it should give you the reason why this is not working. See below:
- SSL Port is blocked
- Certificate does not exist in the keystore/truststore. Check it exists by running
keytool -list -v -keystore keystore.jks
- Certificate chain/Root-intermediate cert is missing
- Server alias/dns entry for this catalog does not exist in the certificate
keytool -list -v -keystore keystore.jks -alias mydomain
mydomain should be replaced with dwpcatalogFQDN
- Firwewall/Loabalancer settings need to be revisited
- Certificate is expired. Check validity
keytool -list -v -alias <alias> -keystore <keystore> -storepass <passwd> alias mydomain | grep "Valid from:"
DWP Catalog checks.
You can make use of attached .ppt file to crosscheck your config vs ppt.
a) Make sure that you used FQDN where it tells you to use FQDN and not the hostname on <installLocation>/sb/rxscripts/bin/setenv.sh
b) Make sure that you used hostname where it tells you to use hostname and not FQDN, in <installLocation>/sb/env/set_script_variables.sh
c) Do not use localhost in the files that we just mentioned above.
d) Make sure that you update the below section on <installLocation>/jetty/etc/jetty-http.xml and put the right/actual keystore and trusttore.
<New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath">/opt/bmc/digitalworkplace/certs/keystore</Set>
<Set name="KeyManagerPassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
<Set name="KeyStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
<Set name="TrustStorePath">/opt/bmc/digitalworkplace/truststore/cacerts</Set>
<Set name="TrustStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
<Set name="IncludeCipherSuites">
e) Make sure ar.conf has below values
- Jetty-Port: 443
- Jetty-Protocol: HTTPS
f)If any changes, restart dwpcontroller and confirm this is now working by running below on another terminal:
netstat -tulpn | grep LISTEN | grep 443; lsof -i :443
You should be able to see something like
[root@xxxx work]# netstat -tulpn | grep LISTEN | grep 443
tcp6 0 0 :::443 :::* LISTEN 17353/java
[root@xxxxxxxx]# lsof -i :443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 17169 root 16u IPv6 4871373 0t0 TCP localhost:41076->localhost:https (CLOSE_WAIT)
java 17169 root 17u IPv6 4871157 0t0 TCP localhost:41070->localhost:https (CLOSE_WAIT)
java 17169 root 18u IPv6 4872364 0t0 TCP localhost:41082->localhost:https (CLOSE_WAIT)
java 17169 root 19u IPv6 4872436 0t0 TCP localhost:41122->localhost:https (CLOSE_WAIT)
java 17353 root 32u IPv6 4855123 0t0 TCP *:https (LISTEN)
And you can also confirm these processes are running by taking the PID and using below command
[root@xxxxx]# ps -aux | grep 17169
[root@xxxxxxx]# ps -aux | grep 17169;ps -aux | grep 17353
root 17169 0.2 0.7 6910620 118908 pts/1 Sl 12:42 0:08 /apps/java/bin/java -jar /apps/dwpc/bin/healthcheck-1.0-shaded.jar /apps/dwpc/bin/monitor.properties
root 27997 0.0 0.0 112716 972 pts/1 S+ 13:32 0:00 grep --color=auto 17169
Jetty <DWPCatalogInstallDir>/sb/jetty/work/jersey.0.0.log should show the below entries
cat jersey.0.0.log | grep -i ssl
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:426)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:320)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:158)
<DWPCatalogInstallDir>/sb/rxscripts/trace.log will give you details too
./sb/rxscripts/trace.log:1:== Info: About to connect() to xxxxxx port 443 (#0)
./sb/rxscripts/trace.log:3:== Info: Connected to xxx (127.0.1.1) port 443 (#0)
./sb/rxscripts/trace.log:8:== Info: subject: CN=cxxxx
./sb/rxscripts/trace.log:11:== Info: common name: xxxxx
./sb/rxscripts/trace.log:12:== Info: issuer: CN=xxxxx
./sb/rxscripts/trace.log:19:00e0: Host: xxxxxx
./sb/rxscripts/trace.log:58:== Info: Added cookie _cacheId="89" for domain xxxx, path /, expire 0
Additional logs can be checked too - for this you will need to add this option jvm.option.24=-Djavax.net.debug=ssl to /apps/dwpc/bin/arserverd.conf - this requires a dwpcontroller restart
- startup_console.log
- armonitor.log
After performing SSL configuration in DWPC server, the dwpcontroller has not come up. It is stuck at Health status check code: 503
We see below error in startup_console.log
INFO - OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. INFO - Error: Could not find or load main class "-Dorg.apache.activemq.broker.BouncyCastlePosition=100 " INFO - Caused by: java.lang.ClassNotFoundException: "-Dorg.apache.activemq.broker.BouncyCastlePosition=100 " INFO - Process Stopped with exit value: 1 INFO - Process [ ARSERVER ] stopped with exit value - [ 0 ]. INFO - Process [ ARSERVER ] stopped with exit value - [ 0 ]. INFO - ProcessMonitor starting process [ ARSERVER ]. INFO - Server Statup Notification Port [SERVERPORT] :42229 INFO - OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. INFO - Error: Could not find or load main class "-Dorg.apache.activemq.broker.BouncyCastlePosition=100 " INFO - Caused by: java.lang.ClassNotFoundException: "-Dorg.apache.activemq.broker.BouncyCastlePosition
In the arserverd.conf file the below line has trailing space. After removing the same, the dwpcontroller has been able to come up.
Before:
Dorg.apache.activemq.broker.BouncyCastlePosition=100 "
After:
Dorg.apache.activemq.broker.BouncyCastlePosition=100 "
Now, both servers should be able to communicate to each other with no issues.
For user group sync issues not working over SSL,
KA 000133376 DWP Catalog - User Group sync script doesn't work with SSL
For AD over SSL issues
KA 000164451 DWP Catalog - Configuring AD Connector on DWP Catalog. Includes AD over SSL