Background: For clustered environment when there is more than one server node that runs SEARCH module (this is typical for multi-node deployment), there is a possibility that Elastic Search instances on those nodes would not be able to form a single Elastic Search Cluster as expecting.
Elastic Search uses Zen Discovery framework underneath that provides functionality to locate other Elastic Search nodes in local network in order to form a cluster. Zen Discovery uses a MULTICAST mechanisms by default to announce node availability. However, MULTICAST could be an additional option provided by network router and may not be available in all networks. In this situation, an Elastic Search instance on one node may not be able to detect presence of an Elastic Search instance on other node. As a result, each instance will form its own single-node cluster isolated from other nodes.
A workaround for this problem might be switching from Elastic Search discovery from MULTICAST to UNICAST. In order to do that, a CDP administrator will need to manually specify addresses of other nodes on every single container server startup script (i.e. start.sh and/or /etc/init.d/apprtx_server) within the cluster:
Add the following system properties (before "-jar lib/container-server-XXXXX.jar") as a single line: