After trying to setup TLS connection on the SMTP Prevent, the connection fails and in the logs you see the following errors:

SmtpPrevent_operational0.log:

[SEVERE] (SMTP_MESSAGE.5304) Failed to decrypt outgoing message (tid=2b cid=11 local=<IP Address>:25 remote=<IP Address>:34642 reason=Unsupported record version Unknown-23.3)

[INFO] (SMTP_CONNECTION.5202) Sender connection error (tid=2b cid=11 local=<IP Address>:25 remote=<IP Address>:34642 reason=javax.net.ssl.SSLException: Unsupported record version Unknown-23.3)

SEVERE: (SMTP_CONNECTION.5208) TLS handshake failed (tid=34 cid=11 local=<IP Address>:25 remote=<IP Address>:41059 reason=no cipher suites in common)

SEVERE] (SMTP_CONNECTION.5208) TLS handshake failed (tid=36 cid=71 local=<IP Address>:25 remote=<IP Address>:57144 reason=Unrecognized SSL message, plaintext connection?) 

RequestProcessor0.log

SEVERE: (SMTP_MESSAGE.5304) Failed to decrypt outgoing message (tid=3b cid=1 local=<IP Address>:25 remote=<IP Address>:40599 reason=Unsupported record version Unknown-23.3)
com.vontu.mta.rp.ESMTPRequestProcessorThread _handleIOException

The TLS connection is failing because the wrong ciphers were used to create the certificates.

By default, the keytool generates DHE-DSS cipher suites as described in the Symantec_DLP_Email_Prevent_MTA_Integration_Guide.

In this case, the ciphers needed were RSA, so the following command had to be used to force the RSA ciphers:

keytool -genkeypair -alias < SMTPPreventHostName > -dname "CN=<SMTPPreventHostName>, OU=Corporate, O=<CompanyName>, L=<location>, S=<State>, C=<country>" -keyalg RSA -validity <NumberOfDays> -keysize 2048 -keystore C:\Vontu\Protect\keystore\prevent.ks

Note the -keyalg RSA argument to make the generated keypair use the RSA ciphers.