The docs show that TLS setup is explicitly only documented with forwarding mode.

Reflection mode is only documented in non-TLS mode. 

TLS is supported within both: reflecting and forwarding mode.

You have to make sure you are following the documentation steps exactly for the implementation in reflecting mode. With a simple terminology change (replacing “downstream” or “next hop” MTA with something more general like “receiving MTA”), the existing instructions also work for reflecting-mode deployments.  In other words, you need to follow the same steps for generating/importing/exporting certificates regardless of reflecting or forwarding mode.

From an integration standpoint any Prevent in reflect mode should be as close to the MTA as possible.

Also, since during the communication the next-hop MTA would send the initial MTA’s response back to themselves I can see that this causes confusion if the originating MTA gets as a response himself but has a different certificate.

Please note:

- The certificate has to be in PEM format
- You have to import the Public Key from the downstream (forward) MTA into the SMTP Prevent keystore in order for it to work
- You may also want to dump the keystore content to make sure that the certificate is imported.

This can be done via the following command

$ keytool -v -list -keystore my.keystore -storepass thepass

My.keystore being the keystore file and thepass being the password for the keystore file.

Please see:

Changing the Network Prevent for Email Server Keystore Password

Please Note

Beginning in version 10.x, Network Prevent (Email) supports the STARTTLS extension to the

SMTP protocol. Network Prevent (Email) now attempts to authenticate MTA connections in

response to the STARTTLS extension.

This feature can cause a Network Prevent (Email) server to log authentication failures

immediately after an upgrade, because the required certificates have not yet been imported to

the correct servers in the proxy chain.

Workaround: If you experience authentication errors after upgrading, you can temporarily

disable TLS authentication to restore the pre-v10.x behavior. See the post-upgrade tasks in

the Symantec Data Loss Prevention Upgrade Guide for instructions.

See the Symantec Data Loss Prevention MTA Integration Guide for Network Prevent (Email) for

information about generating and importing the certificates that are required to support TLS

authentication.