How to setup TLS encryption and what logs to set if further troubleshooting is required
Make sure that the following settings are set for proper logging on the SMTP Prevent side we would need for reviewing| Set the log levels on SMTP Prevent Within the SymantecDLP/Protect/config directory adjust the file RequestProcessorLogging.properties and set or assign the following value
Also, we had issues with Iron Mail and others where the following worked
(1) Please verify that the wording of "STARTTLS" has presented in the "RequestProcessor.AllowExtensions" whitelist.
If not, add it in, and recycle the SMTP prevent server and try it again. To edit, log into the DLP Console as Administrator and go to System > Servers > Overview > Click on the SMTP Prevent Server > Click on 'Server Settings'
(2) If above still not working, then the problem may relate to the format of the certificate that imported from MTA.
Proper X509 certificates need to be installed on prevent in order for this to work securely. (In this POA - ie. trustedcert.pem) .pem file -- Privacy Enhanced Mail Certificate
Follow steps below for TLS setup
A. Down Stream MTA
1. Generate a new public/private key pair: openssl req -new -key ca.key -out ca.csr 2. Generate a .pem file by signing a certificate request with a CA key: openssl x509 -trustout -signkey ca.key -days 365 -req -in ca.csr -out trustedcert.pem
B. Enable TLS in SMTP Prevent via Enforce
1. To "Configure" a. Set Keystore Password b. Select "Reflect" mode 2. To "Server Settings" a. RequestProcessor.AllowExtensions add "STARTTLS" to the whitelist b. RequestProcessor.ServerSocketPort (as IronMail required) c. RequestProcessor.MTAResubmitPort (as IronMail required)
C. SMTP Prevent (Windows)
1. Download the (downstream) MTA's cert (e.g. trustedcert.pem) to SMTP Prevent's local directory (say c:\mtacert) 2. Create a keystore and self-signed cert a. Use the keytool from c:\SymantecDLP\jre\bin folder b. Make sure 'protect' has read permissions on the keystore c. Run - keytool -genkey -alias smtp_prevent -keystore c:\SymantecDLP\Protect\keystore\prevent.ks 3. Import downstream MTA trusted certificate (.pem format) into a Java Keystore a. Run - keytool -importcert -keystore c:\SymantecDLP\Protect\keystore\prevent.ks -alias MTA -file c:\mtacert\trustedcert.pem
D. Recycle the SMTP Prevent server, and test the TLS mailing process and collect the logs send them back for further review by Symantec engineering.
In cases where Exchange and Message Labs* have been involved, the following seemed to work.
Messages are coming from and Exchange 2007 server to SMTP Prevent (v10) and then forwarding mail to Message Labs. The customer has a requirement that mail be transmitted via TLS until it gets to Message Labs (from there it is opportunistic). We were experiencing problems where mail would queue on the Exchange servers every time we tried to route mail through Prevent. At first it was a certificate issue and I won’t go into the details of that one. However, over the last couple of days we seemed to be completing the TLS handshake, but the connection would get dropped.
The MTA integration guide states very clearly that the Prevent server has to have the Public key certificate from the downstream MTA in its keystore. The Message Labs guys were saying that we DID NOT need this because whenever it negotiates a TLS session, it sends its public key as part of the handshake. For a long time, we relied on this…however after many attempts to get this to work without success, the team decided to intercept the return message and public key from Message Labs and then import it into the keystore on Prevent. BOOM, it works. Why?
Apparently, Prevent doesn’t trust ANYBODY, so it validates the public key that it has in its own keystore against the public key that gets transmitted to it when it starts the TLS handshake. If they don’t match, Prevent terminates the connection. This is not, apparently part of the RFC, but seems to be the way it works.
Bottom line: We HAVE TO import the Public key from the downstream (forward) MTA into the Prevent keystore in order to make TLS work.
Note: Within Sendmail environments ( Sentrion ), the following settings have to be disabled in the Sendmail.cf file.