TSCO is failing to send e-mail messages either associated with Diagnostic Alerts, associated with some or all reports, or both. Key considerations in relation to debugging this problem include: (1) Is TSCO able to send any e-mail messages? So, is the problem limited to just specific reports or is TSCO unable to send any e-mail? (2) Has TSCO been able to send SMTP e-mails but has recently stopped being able to send e-mails? This is useful to know because if it has been working it would make the general scenario that SMTP isn't properly configured in TSCO less likely. (3) SMTP errors are often only reported in the TSCO logs ($CPITBASE/scheduler/log/cpit.log most commonly) and don't make it up to the Diagnostic Alert level. Thus, when debugging SMTP problems it is generally necessary to review the TSCO logs. |
Initial Problem AnalysisWhen TSCO sends e-mail it will directly connect to the defined SMTP server -- so you won't see any messages about the e-mail being sent from TSCO in the /var/log/messages file since TSCO isn't using sendmail (or a sendmail equivalent) running on the Linux machine) to send the e-mail. It is instead connecting directly to the defined SMTP server and sending the e-mail that way.The first place to check is the $CPITBASE/scheduler/log/cpit.log to see if there is a more detailed error message associated with the e-mail failing to be sent. For example, you might see a message like this: 2017-07-27 10:07:37,517 INFO [ReportThread-0]- [ReportThread-0] Report 70 saved as 70/report_70_1501167600398/Production Weekly - TSCO 2017-07-27.html In that case the problem is that the SMTP server is rejecting the report e-mail because the message is too large. Depending on the report configuration you may be able to reduce the size by decreasing the number of attachments (if you've selected to include the report output in multiple formats). You could look at the report output in TSCO and see how big the attachment is that TSCO was trying to e-mail. Maybe the report output is extremely large and something can be done to reduce the size of the report. Beyond that I think the only option would be to discuss this with your SMTP Administrators to see if there was another way (possibly a less restrictive SMTP server for internal communications) that could be used to send the large e-mail message with the attached reports. Also, by any security policy the SMTP server should have the App Server as a trusted client. Connection refused message: java.net.ConnectException: Connection refusedIn some cases we may see these kind of error because the protocol expected by the STMP error is not the same as the one initiated by the TSCO console. In that case it might happen that TSCO console initiates an SSL connection on port 25 and this is incorrect. The correct port to start a secured connection should be 465, this analysis works backwards. Here is an example of an error with connection refused message:2018-11-10 09:30:06,720 ERROR [taskid=18]- BCO_SCH_ERR016: Error sending mail: Could not connect to SMTP host: [SMTP host], port: 465 Testing basic SMTP connectivityIf the problem is that TSCO isn't able to send any e-mail messages the problem may be that the SMTP server is rejecting the connection (possibly because it requires authentication or encryption). One can test this by trying to connect to your SMTP server directly from the command line using the 'telnet' command and using the parameters you've specified in your TSCO configuration if you are just using plain unencrypted and unauthenticated SMTP on port 25. So, the command you are going to run is this: telnet smtp.domain.com 25 Then you are going to enter the following within that session (one line at a time): HELO MAIL FROM: noreply.BCO@server.domain.com RCPT TO: user@domain.com DATA Test message. . QUIT If you don't receive any errors from the above then TSCO should also be able to send e-mail. Here is an example of what that session should look like: > telnet smtp.domain.com 25 Trying 192.168.100.1... Connected to smtp.domain.com. Escape character is '^]'. 220 hostname.domain.gov Microsoft ESMTP MAIL Service, Version: 7.5.7601.17514 ready at Thu, 27 Jul 2017 17:23:49 -0500 HELO 250 hostname.domain.gov Hello [192.168.1.1] MAIL FROM: noreply.BCO@server.domain.com 250 2.1.0 noreply@hostname.com....Sender OK RCPT TO: user@domain.com 250 2.1.5 user@domain.com DATA 354 Start mail input; end with <CRLF>.<CRLF> Test message. . 250 2.6.0 <hostname.domain.comLstP000086f0@hostname.domain.com> Queued mail for delivery QUIT 221 2.0.0 hostname.domain.com Service closing transmission channel Connection closed by foreign host. If the problem is just a general SMTP access problem the above test will generate an error. Depending on the error we'll likely need to engage your SMTP Administrators to understand what the requirements are to connect to your SMTP server to send e-mail. If you see a high number of sendmail processes generated by the cpit user which starts the services, it is very likely not caused directly by TSCO, because TSCO is not using sendmail. it is more likely that the user is a member of the cron group and the watchdog.sh script is triggered by cron. And is is very likely that cron is configured to send mails every time a cron job run. You should point check the /varlog/cron and /var/log/mailsend and run the mail command under the user to see more details. |