Following instructions are also available in the Administration Guide during an attempt to establish a 3rd party certificate for the Enforce UI.
But this error, ""public keys in reply and keystore don't match", appears when running the last step which is importing the CA back into the original keystore file in tomcat.
keytool -import -alias tomcat -keystore .keystore -trustcacerts -storepass <your password> -file <your filename>.cer
This resulted from a mismatch between the hash of the requested cert, and the originally generated (signed) certificate.
This means the details in the Private key (original to the self-signed certificate created) do not match the Public Key for which the customer's CA cert has been created.
-dname "cN=<your CN>, O=<your organization>, Ou=<your OU>, L=<your location>, S=<your state>, C=<your country>"
Ensure that CSR includes correct details for the certificate: