How to setup TLS encryption, and what logs to set if further troubleshooting is required.

Make sure that the following settings are set for proper logging on the SMTP Prevent server.
Set the log levels on the Network Prevent for Email detection server
Within the ..\Protect\config directory adjust the file RequestProcessorLogging.properties and set or assign the following value

com.vontu.mta.log.SmtpOperationalLogHandler.level = FINEST
java.util.logging.FileHandler.level = FINEST 
com.vontu.mta.rp.level = FINEST

Then restart the detection server service.

(1) Please verify that the wording of "STARTTLS" has presented in the "RequestProcessor.AllowExtensions" whitelist.

If not, add it in, and recycle the SMTP prevent server and try it again.
To edit, log into the DLP Console as Administrator and go to:
System > Servers > Overview > Click on the SMTP Prevent Server > Click on 'Server Settings'

(2) If above still not working, then the problem may relate to the format of the certificate that imported from MTA.

Proper X509 certificates need to be installed on prevent in order for this to work securely.
(In this POA - ie. trustedcert.pem) .pem file -- Privacy Enhanced Mail Certificate.

Follow steps below for TLS setup:

A. Down Stream MTA.

1. Generate a new public/private key pair:

openssl req -new -key ca.key -out ca.csr

2. Generate a .pem file by signing a certificate request with a CA key:

openssl x509 -trustout -signkey ca.key -days 365 -req -in ca.csr -out trustedcert.pem

B. Enable TLS in SMTP Prevent via Enforce.

1. To "Configure"
     a. Set Keystore Password
     b. Select "Reflect" mode
2. To "Server Settings"
     a. RequestProcessor.AllowExtensions add "STARTTLS" to the whitelist
     b. RequestProcessor.ServerSocketPort (as IronMail required)
     c. RequestProcessor.MTAResubmitPort (as IronMail required)

C. Detection Server Email Prevent (Windows).

Locations of Keytool vary with version of DLP and OS of your servers:

Windows:

• DLP 16.0 RU1, 16.0 and 15.8: C:\Program Files\AdoptOpenJRE\jdk8u<version>-jre
• DLP 15.7: C:\Program Files\Symantec\DataLossPrevention\ServerJRE

Linux:

• DLP 16.0 RU1, 16.0 and 15.8: /opt/AdoptOpenJRE/jdk8u<version>-jre
• DLP 15.7: /opt/Symantec/DataLossPrevention/<version>

1. Download the (downstream) MTA's cert (e.g. trustedcert.pem) to SMTP Prevent's local directory (say c:\mtacert)
2. Create a keystore and self-signed cert
     a. Use the keytool from DLP (see keytool locations below)
     b. Make sure 'protect' has read permissions on the keystore
     c. Run command

Windows: keytool -genkey -alias smtp_prevent -keystore C:\ProgramData\Symantec\DataLossPrevention\DetectionServer\[YourVersion]\keystore\prevent.ks

Linux: keytool -genkey -alias smtp_prevent -keystore /var/Symantec/DataLossPrevention/DetectionServer/[YourVersion]/keystore/prevent.ks

3. Import downstream MTA trusted certificate (.pem format) into a Java Keystore as below:

Windows: keytool -importcert -keystore c:\ProgramData\Symantec\DataLossPrevention\DetectionServer\[YourVersion]\keystore\prevent.ks -alias MTA -file c:\mtacert\trustedcert.pem

Linux: keytool -importcert -keystore
/var/Symantec/DataLossPrevention/DetectionServer/[YourVersion]/keystore/prevent.ks -alias MTA -file /var/location of MTA cert/trustedcert.pem

D. Recycle the SMTP Prevent server, and test the TLS mailing process and collect the logs send them back for further review by Symantec engineering.

In cases where Exchange and Message Labs have been involved, you may need to try the following:

Messages are coming from and Exchange 2007 server to SMTP Prevent (v10) and then forwarding mail to Message Labs. There is a requirement that mail be transmitted via TLS until it gets to Message Labs (from there it is opportunistic).

The MTA integration guide states that the Prevent server has to have the Public key certificate from the downstream MTA in its keystore. Message Labs sends its public key as part of the handshake.

Email Prevent validates the public key that it has in its own keystore against the public key that gets transmitted to it when it starts the TLS handshake. If they don’t match, Email Prevent terminates the connection. 

You must import the Public key from the downstream (forward) MTA into the Prevent keystore for TLS to work.

Note:
Within Sendmail environments ( Sentrion ), the following settings have to be disabled in the Sendmail.cf file.

ServerSSLOptions=SSL_OP_NO_SSLv2 Client
SSLOptions=SSL_OP_NO_SSLv2

Otherwise sendmail will not conform fully to RFC 2246. Since SMTP Prevent is a TLSv1 compliant SMTP Proxy it would fail to establish a connection since SSLv2 is required to negotiate TLSv1.