...
Figure 1 - Orchestrate GeoDistributed Edge Applications
> **NOTE**: A 'composite application' is a combination of multiple applications with each application packaged as a Helm Chart. Based on the deployment intent, various applications of the composite application get deployed at various locations, and get replicated in multiple locations.
...
For a given use case, the number of K8s clusters (edges or clouds) could be in tens of thousands. The number of composite applications that need to be managed could be in the hundreds. The number of applications in a composite application could be in the tens. The number of micro-services in each application of the composite application can be in the tens. Moreover, there can be multiple deployments of the same composite application for different purposes. To reduce this complexity, all of these operations can be automated. A Multi-Edge and Multi-Cloud distributed application orchestrator enables one-click deployment of the composite applications and makes a one simple dashboard to know the status of the composite application deployment at any time possible.
...
EMCO Terminology and Concepts
| Term | Description |
|---|---|
| Cluster Provider | The cluster provider is the entity that owns and registers clustesrs with EMCO. |
| Cluster | Clusters are registered with EMCO. Access to a given cluster may be via kubeconfig or via one of the supported gitOps methods. |
| Project | The project provides a means of grouping collections of applications. |
| Multiple applications may exist for any project. Projects allow for defining applications with different tenants - i.e. a given tenant is associated with one or more projects that are distinct from another tenants projects. | |
| Logical Cloud | The logical cloud is an EMCO concept that groups a number of clusters together and provides a common set of namespace, permissions and quotas. Logical clouds are defined under projects and are the target deployment destination of EMCO Composite Applications. Placement scheduling will ultimately determine which cluster(s) in a logical cloud any specific component of a composite application will deployed to. |
| Composite Application | The composite application is a combination of multiple applications that work together. Each application in the composite application is a Helm chart. Through the use of placement intents, EMCO allows different applications in the composite application to be deployed (and replicated as necessary) to different sets of clusters. For example, a composite application composed of a user facing application along with a database application could be deployed by EMCO such that the user facing application is deployed to many edge clusters while the database application is deployed on a centralized cluster. |
| Deployment Intents | The deployment intents are the various placement and action intents that have been defined to control the deployment of an EMCO composite application. Placement and action intents are defined and handled by various EMCO controllers. Different composite application deployments may use differents sets of placement and action intents. EMCO does not expect the Helm charts which are onboarded and comprise a composite application to be |
edited or modified. Instead, any customizations or additional resources required to ensure the composite application operates correctly when deployed are defined by the set of deployment intents. | |
| Placement Intents | Placement intents are used to identify to which clusters the resources of each application of a composite application are to be deployed. Each composite application is required to define a set of generic placement intents. Additional placement intents, for more specialized purposes such as hardware capabilities, latency, etc., may be defined as well. The additional placement intents will refine the results of the generic placement. |
| Action Intents | Action intents are provided by special action controllers and are used to perform specific customizations to the EMCO composite application before it is deployed. After EMCO has processed the placement intents for a specific composite application deployment, the action intents are processed. The action intents may do anything from customizing existing application resources to adding new resources that need to be deployed. EMCO provides many action controllers and additional controllers may be |
| easily written and used with EMCO. | |
| EMCO Database | The EMCO data base (currently MongoDB) is used primarily to store the resources onboarded by the EMCO REST APIs (e.g. composite applications, cluster data, all the various deployment intents). |
| EMCO AppContext | The EMCO AppContext is a data store (currently `etcd`) that is used by EMCO to prepare and manage the resources that will be deployed to edge clusters. When EMCO is preparing a composite application for deployment by processing the various deployment intents, the results are prepared in the AppContext. Once all placement and action intents have been handled, the AppContext contains the set of resources that need to be applied to the set of target clusters. The EMCO `rsync` ('resource synchronizer and status collector') microservice then deploys the resources to the edge clusters. The AppContext is also used to collect status information back from the edge clusters as detected by `rsync`. |
EMCO Architecture
The following diagram depicts a high level overview of the EMCO architecture.
...
- Cluster Registration Controller registers clusters by cluster owners.
- Distributed Application Scheduler provide simplified and extensible placement.
- Network Configuration Management handles creation and management of virtual and provider networks.
- Hardware Platform Aware Controller enables scheduling with auto-discovery of platform features/ capabilities.
- Distributed Cloud Manager presents a single logical cloud from multiple edges.
- Secure Mesh Controller auto-configures both service mesh (ISTIO) and security policy (NAT, firewall).
- Secure WAN Controller automates secure overlays across edge groups.
- Resource Syncronizer Synchronizer manages instantiation of resources to clusters.
- Monitoring covers distributed applications.
...
The Distributed Application Scheduler supports operations on a deployment intent group resource to instantiate the associated composite application with any placement and action intents performed by the registered placement and action controllers. The basic flow of lifecycle operations on a deployment intent group after all the supporting resources have been created via the APIs are:
| Operation | Description |
|---|---|
| approve | marks that the deployment intent group has been approved and is ready for instantiation. |
| instantiate | the Distributed Application Scheduler prepares the application resources for deployment, and applies placement and action intents before invoking the Resource Synchronizer to deploy them to the intended remote clusters. In some cases, if a remote cluster is intermittently unreachable, the instantiate operation will retry the instantiate operation for that cluster until it succeeds. |
| status | (may be invoked at any step) provides information on the status of the deployment intent group. |
| terminate | terminates the application resources of an instantiated application from all of the clusters to which it was deployed. The terminate operation will cause the instantiate operation to complete (i.e. fail), before the termination operation is performed. |
| stop | In some cases, if the remote cluster is intermittently unreachable, the Resource Synchronizer will continue retrying an instantiate or terminate operation. The stop operation can be used to force the retry operation to stop, and the instantiate or terminate operation will complete (with a failed status). In the case of terminate, this allows the deployment intent group resource to be deleted via the API, since deletion is prevented until a deployment intent group resource has reached a completed terminate operation status. |
| update | After a deployment intent group has been instantiated, it may be updated (applications may be modified and/or various deployment intents may be added or modified). The update operation is used to deploy the changes to the running / deployed composite application. |
| rollback | After a deployment intent group has been instantiated and updated one or more times, it is possible to rollback the composite application to a previously deployed version. |
Refer to [EMCO Resource Lifecycle Operations] for more details.
...
The Network Configuration Management microservice supports operations on the network intents of a cluster resource to instantiate the associated provider and virtual networks that have been defined via the API for the cluster. The basic lifecycle operations flow on a cluster, after the supporting network resources have been created via the APIs, are:
| Operation | Description |
|---|---|
| apply | the Network Configuration Management microservice prepares the network resources and invokes the Resource Synchronizer to deploy them to the designated cluster. |
| status | (may be invoked at any step) provides information on the status of the cluster networks. |
| terminate | terminates the network resources from the cluster to which they were deployed. In some cases, if a remote cluster is intermittently unreachable, the Resource Synchronizer may still retry the instantiate operation for that cluster. The terminate operation will cause the instantiate operation to complete (i.e. fail), before the termination operation is performed. |
| stop | In some cases, if the remote cluster is intermittently unreachable, the Resource Synchronizer will continue retrying an instantiate or terminate operation. The stop operation can be used to force the retry operation to stop, and the instantate or terminate operation will be completed (with a failed status). In the case of terminate, this allows the deployment intent group resource to be deleted via the API, since deletion is prevented until a deployment intent group resource has reached a completed terminate operation status. |
Distributed Cloud Manager - `dcm`
...
Whenever rsync starts or restarts, it checks for active AppContextIDs in the "activecontext" area of the etcd using the prefix key: `/activecontext`.
If it finds an active AppContextID, it creates the AppContextData and starts the main thread for handling the pending AppContextIDs which were recorded as active contextIDs.
Rsync state machine
| Event | Valid Starting Current State | Desired State | Current State | Error State |
|---|---|---|---|---|
| InstantiateEvent | Created, Terminated, TerminationFailed, Instantiated, Instantiating, InstantiationFailed, Update, Updating, UpdateFailed | Instantiated | Instantiating | InstantiateFailed |
| TerminateEvent | Terminating, TerminationFailed, Instantiated, Instantiating, InstantiationFailed, Updated, Updating, UpdateFailed | Terminated | Terminating | TerminationFailed |
| UpdateEvent | Created, Updated | Updated | Updating | UpdateFailed |
| UpdateDeleteEvent | Instantiated | Instantiated | Instantiating | InstantiateFailed |
Status Monitoring and Queries in EMCO
...