Luminate is a Software as a Service platform that allows corporate IT organizations to provide secure connectivity to corporate applications and services for designated audiences. Luminate allows doing the above without exposing internal networks/datacenters to risks associated with network access and provides unparalleled visibility and governance into activities that are performed by the accessing parties.

In order to get started with Luminate, one needs to configure the system to provide access as described in the below diagram:

Basic_Luminate_Deployment.PNG

In the diagram above, users can access published applications via the Luminate cloud service, as these applications have a Luminate Connector component deployed in their datacenter. A separate guide explains the steps that need to be taken in order to configure applications for access via Luminate.

This guide explains how to deploy Luminate Connectors in an Amazon Web Services VPC, while allowing AWS to manage their availability and redundancy, without the need to manage any dedicated EC2 Instances or other components. 

General Description

In order to ensure continuous access to the applications / services via the Luminate platform, Connectors need to be up and running (and should be able to access the target applications/services) inside the customer organization's Amazon Web Services VPC.

The below diagram depicts the most basic deployment of Luminate Connectors in the Elastic Container Service:

Basic_AWS_ECS_Connector_Deployment.png

The diagram shows how the two main goals of the orchestrated deployment are achieved:

  • Availability - Elastic Container Service (ECS) makes sure that the Container "Service" is constantly up and running (and is doing so in different Availability Zones)
  • Redundancy - Elastic Container Service (ECS) distributes different "Services" among Container Instances located in different Availability Zones.

For more information about the Amazon Elastic Container Service please access the AWS Documentation.

The deployment described in the diagram above consists of the following AWS entities:

  • ECS Cluster - An entity representing a logical cluster of EC2 instances that are hosting various services delivered in a Docker Container format. The cluster takes care both of making sure that there are healthy ECS Instances ready to host the services in various availability zones (based on the cluster configuration) and of orchestrating execution of various services on different instances.
  • ECS Instances (Container Instances) - EC2 Instances based on a special image prepared by AWS and managed by the ECS Cluster. The ECS Agent running on these instances takes care of automatically launching tasks based on the cluster definitions.
  • ECS Task Definitions - Logical entities describing the workload, consisting of: Docker Image, Entry Point (if different from default), Environment Variables, Persistent Volumes, Resource Constraints required for the running container, etc... 
  • ECS Services - Logical entities describing the required state of execution for various task definitions. Specifically, how many instances should run, how they should be distributed across various ECS Instances, etc... The ECS agent is using these definition, along with the Task Definitions
  • ECS Running Tasks - Logical entities describing the currently running containers (placed on one of the ECS Instances belonging to the ECS Cluster) based on one of the Task Definitions and associated with a defined Service.

 Below diagram describes the relationship between these entities:

ECS_Entities_Relationship.PNG

During the operation of the system, ECS Cluster creates Running Tasks (creates new Docker Containers running on one of ECS Instances) according the to definitions and requirements defined in Services. The creation of new containers is subject to the Task Definitions.

Each Luminate Connector defined in the Deployment view of the Luminate Administration Web Portal will be represented by a unique Task Definition and Service. Each time it is being run, a Running Task will be created by the ECS Agent. The task definition will contain all the relevant environment variables (see below) required to launch a Connector successfully.

Depending on various conditions, the ECS Cluster can stop currently running tasks / ECS Instances, spawn new ECS Instances and deploy running tasks on them. This is a very important point, as, when the Running Task representing one of Luminate Connectors is terminated and then re-created, we need to make sure that the identity of the connector remains. In order to achieve this, we will use Volumes and Mount Points when declaring Task Definitions in the below steps. The Volumes defined in ECS Task Definitions are stored on the file systems defined on the ECS Instances. In order to allow the system to launch Connectors on a different instance, the administrator needs to define a shared file system (in our case we will use AWS Elastic File System (EFS)) or synchronize the content of the volumes in any other way (for example, using S3 Buckets).

Step-by-Step Guide

  1. Define Connectors in the Luminate Administration Web Portal: As this document assumes that our goals are High-Availability and Redundancy of Connectors, the amount of defined connectors should not exceed the amount of ECS Instances in the ECS Cluster. General recommendation is to have either 2 or 3 instances, each running in a different Availability Zone. (Some AWS Regions have 2 Availability Zones, others have 3, therefore the amount of Connectors and ECS Instances depends on the AWS Region used.) The process of defining Connectors is described in this article.
  2. Define the ECS Cluster in Your Amazon Web Services VPC: Step-by-step guide for defining an ECS Cluster is provided in this AWS Document. If the system is planned to be able to "survive" destruction and recreation of ECS Instances, an external storage should be connected to each of the automatically created machines. This tutorial and this blog from AWS describe how to use Elastic File System with ECS clusters to achieve this. When creating the cluster it is important to note the following parameters:
    1. Provisioning Model: Since the Connector services are required to run in order to provide connectivity to corporate applications / services, the desired model is On-Demand instances, and not Spot Instances.
    2. EC2 Instance Type: The system requirements for a connector are not demanding. The size of the instance should depend on the expected traffic.
    3. Number of Instances: The recommendation is to create an instance for each Availability Zone.
    4. Key Pair: It is extremely important to pick a key pair that can later be used to connect to the instances, in rare cases of advanced troubleshooting scenarios.
  3. Create Task Definitions for each Connector: The general description of task definitions can be found in this AWS Document. In this specific case, it is important to focus on the following elements:
    1. Task Name: The name of the task definition should be related to the name of the connector defined in the Luminate Administration Portal. 
    2. Network Mode: As the nature of Connector is to communicate with entities outside of the host it is running on, Host Mode is the right choice.
    3. Volumes: At least one volume needs to be created. Its Source Path depends on the chosen storage volume. If an external file system is mounted, it is better to create is under /mnt/<Mount Point>, otherwise there is no significance to its placement.
    4. Container Definitions
      1. Image: The image should refer to a repository containing images of Luminate Connector. If a public image is used (the common case), this field should contain luminate/connector or luminate/connector:specific-tag
      2. Hard / Soft Memory Limits: Defining these is not mandatory. When doing it, one needs to take into consideration the EC2 Instance Types chosen when defining the ECS Cluster, as the orchestrating mechanism will calculate these vs the machine capacity when distributing running tasks.
      3. Essential: It is mandatory to define one container as mandatory. As our task has only one Container Definition, it has to be marked as essential.
      4. Mount Point: It is essential to define a mount point for the source volume defined in the task definition. This is where the Connector will store its identity and the information required to create a secure connection to the Luminate Cloud. The Container Path could be defined to any location, and needs to be used later. For example, /opt/luminate-data.
      5. Port Mappings: Since the Connector doesn't require a service "listening" to an external port, this definition is not mandatory. It could be beneficial to define one port as mapped for all task definitions in order to enforce the ECS Agent to distribute Connectors to different ECS Instances. TCP Port 443 (both on the host and on the container) is recommended for that purpose.
      6. Environment Variables: This is a very important section that should be filled with data coming from the Luminate Administration Web Portal, with an addition of a special environment variable for the usage of the mount point for persistent data.
        1. The content of the environment variables below should be copied:
        2.  Please make sure that you omit the single quotes that appear in the variable values. 
        3. Additional variable should be added - STORAGE_PATH. Its value should be set to a file with distinctive name under the Container Path chosen for the mount point, for example /opt/luminate-data/<connector_name>_storage. If the same paths on the ECS Instances are going to be used as Source Path in different Task Definitions, the file name should differ between various Connector task definitions.
  4. Create Services for each Task Definition: After completing the task definition, choose Actions -> Create Service. Following parameters of the service are important:
    1. Service Name: Should reflect the name of the connector, for the ease of monitoring and troubleshooting.
    2. Number of Tasks:Needs to be 1, in order to make sure that only one instance of each Luminate Connector is running simultaneously.
    3. ELB Type: Needs to be set for 'None', as Connector does not require a load balancer (there are no inbound connections)
    4. Service Auto Scaling: Should be set to 'Do not adjust the service’s desired count'.

A successful definition should end with the following line describing the newly created ECS Cluster in the AWS Console:

ECS_Cluster_for_Connector.PNG

The amount of Services, Running tasks and Container instances should match the ones defined when creating an ECS Cluster. When configured correctly, launching the cluster will also move the relevant connectors in the "Online" state in the Luminate Administration Web Portal:

online_connectors.PNG