Observability
- Todd Malsbary
- Artashes Balabekyan
Overview
The purpose of this page at this time is to capture requirements related to observability of the EMCO services (https://gitlab.com/groups/project-emco/-/epics/7).
Front-ending the services with Istio provides a useful set of metrics and tracing, and adding the Prometheus library provided collectors to each service expands that with other fundamental metrics. The open question is what additional metrics and tracing will be useful to EMCO operators.
Metrics
The following items are based on Prometheus recommendations for instrumentation.
Queries, errors, and latency
Both client and server side are provided by Istio. https://istio.io/latest/docs/reference/config/metrics/
Istio metrics can be customized to include other attributes from Envoy such as subject field of peer certificate. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes
Example PromQL
Service | Type | PromQL | Notes |
|---|---|---|---|
HTTP/gRPC* *The request_protocol label can be used to distinguish among HTTP and gRPC. | Queries | sum(irate(istio_requests_total{reporter="destination",destination_workload=~"services-orchestrator"}[5m])) | inbound |
sum(irate(istio_requests_total{reporter="source",source_workload="services-orchestrator"}[5m])) by (destination_workload) | outbound | ||
Errors | sum(irate(istio_requests_total{reporter="destination",destination_workload=~"services-orchestrator",response_code!~"5.*"}[5m])) / sum(irate(istio_requests_total{reporter="destination",destination_workload=~"services-orchestrator"}[5m])) | inbound | |
sum(irate(istio_requests_total{reporter="source",source_workload=~"services-orchestrator",response_code!~"5.*"}[5m])) by (destination_workload) / sum(irate(istio_requests_total{reporter="source",source_workload=~"services-orchestrator"}[5m])) by (destination_workload) | outbound | ||
Latency | histogram_quantile(0.90, sum(irate(istio_request_duration_milliseconds_bucket{reporter="destination",destination_workload="services-orchestrator"}[1m])) by (le)) / 1000 | P90 | |
Saturation |
Queries, errors, and latencies of resources external to process (network, disk, IPC, etc.)
The prometheus golang library provides builtin collectors for various process and golang metrics: https://pkg.go.dev/github.com/prometheus/client_golang@v1.12.2/prometheus/collectors. A list of metrics provided by cAdvisor is at https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md. Additional K8s specific metrics can be enabled with the https://github.com/kubernetes/kube-state-metrics project.
Example PromQL
Note: some of these require that kube-state-metrics is also deployed.
Pod Resource | Type | PromQL |
|---|---|---|
CPU | Utilization | sum(rate(container_cpu_usage_seconds_total{namespace="emco"}[5m])) by (pod) |
Saturation | sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="emco"}[5m])) by (pod) | |
Errors | ||
Memory | Utilization | sum(container_memory_working_set_bytes{namespace="emco"}) by (pod) |
Saturation | sum(container_memory_working_set_bytes{namespace="emco"}) by (pod) / sum(kube_pod_container_resource_limits{namespace="emco",resource="memory",unit="byte"}) by (pod) | |
Errors | ||
Disk | Utilization | sum(irate(container_fs_reads_bytes_total{namespace="emco"}[5m])) by (pod, device) |
sum(irate(container_fs_writes_bytes_total{namespace="emco"}[5m])) by (pod) | ||
Saturation | ||
Errors | ||
Network | Utilization | sum(rate(container_network_receive_bytes_total{namespace="emco"}[1m])) by (pod) |
sum(rate(container_network_transmit_bytes_total{namespace="emco"}[1m])) by (pod) | ||
Saturation | ||
Errors | sum(container_network_receive_errors_total{namespace="emco"}) by (pod) | |
sum(container_network_transmit_errors_total{namespace="emco"}) by (pod) |
Internal errors and latency
Internal errors should be counted. It also desirable to measure success to calculate ratio.
Totals of info/error/warning logs
Unsure if this is a useful metric.
Any general statistics
This bucket includes EMCO specific information such as number of projects, errors and latency of deployment intent group instantiation, etc. Also consider any cache or threadpool metrics.
Preliminary guidelines:
Distinguish between resources and actions.
Action metrics will record requests, errors, and latency similar to general network requests.
Resource metrics will record creation, deletion, and possible modification.
Metrics will be labeled with project, composite-app, deployment intent group, etc.
For rsync specifically, measure health/reachability of target clusters.
Also, keep in mind this cautionary note from the Prometheus project:
CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.
However note that well-known projects such as Istio and kube-state-metrics appear to disregard this, so further investigation may be needed on the motivations behind this note.
Preliminary metrics
This section contains some of the considerations of the guidelines above applied to the orchestrator service.
The actions of a service can be identified from the gRPC requests and HTTP lifecycle requests:
Service | Action |
|---|---|
orchestrator | approve |
instantiate | |
migrate | |
rollback | |
stop | |
terminate | |
update | |
StatusRegister | |
StatusDeregister |
The requests, errors, and latency can be modeled after Istio's istio_requests_total and istio_request_duration_milliseconds, with an additional action name label.
The resources of a service can be identified from the HTTP resources. The initial labels can be the URL parameters.
Service | Resource | Labels |
|---|---|---|
orchestrator | controller | name |
project | name | |
compositeApp | version, name, project | |
app | name, composite_app_version, composite_app, project | |
dependency | name, app, composite_app_version, composite_app, project | |
compositeProfile | name, composite_app_version, composite_app, project | |
appProfile | name, composite_profile, composite_app_version, composite_app, project | |
deploymentIntentGroup | name, composite_app_version, composite_app, project | |
genericPlacementIntent | name, deployment_intent_group, composite_app_version, composite_app, project | |
genericAppPlacementIntent | name, generic_placement_intent, deployment_intent_group, composite_app_version, composite_app, project | |
groupIntent | name, deployment_intent_group, composite_app_version, composite_app_name, project | |
dcm | emco_logical_cloud_resource | project, name, namespace, status |
clm | emco_cluster_provider_resource | name |
emco_cluster_resource | name, clusterprovider | |
ncm | emco_cluster_network_resource | clusterprovider, cluster, name, cnitype |
emco_cluster_provider_network_resource | clusterprovider, cluster, name, cnitype, nettype, vlanid, providerinterfacename, logicalinterfacename, vlannodeselector | |
dtc | emco_dig_traffic_group_intent_resource | name, project, composite_app, composite_app_version, dig |
emco_dig_inbound_intent_resource | name, project, composite_app, composite_app_version, dig, traffic_group_intent, spec_app, app_label, serviceName, externalName, port, protocol, externalSupport, serviceMesh, sidecarProxy, tlsType | |
emco_dig_inbound_intent_client_resource | name project, composite_app, composite_app_version, dig, traffic_group_intent, inbound_intent, spec_app, app_label, serviceName | |
emco_dig_inbound_intent_client_access_point_resource | name, project, composite_app, composite_app_version, dig, traffic_group_intent, inbound_intent, client_name, action | |
ovnaction | emco_network_controller_intent_resource | name, project, composite_app, composite_app_version, dig |
emco_workload_intent_resource | name, project, composite_app, composite_app_version, dig, network_controller_intent, app_label, workload_resource, type | |
emco_workload_interface_intent_resource | name, project, composite_app, composite_app_version, dig, network_controller_intent, workload_intent interface, network_name, default_gateway, ip_address, mac_address |
The metrics for these resources should capture the state of the resource, i.e. metrics for creation, deletion, etc. (emco_controller_creation_timestamp, emco_controller_deletion_timestamp, etc.) as described in the guidelines. This approach is suggested as it is unclear how to apply metrics capturing resource utilization to these resources.
The status of a deployment intent group deserves special consideration. The suggested approach is to support the labels necessary to execute equivalent queries as shown in EMCO Status Queries. This would enable alerting on the various states of the resources composing a deployment intent group.
Metric | Type | Description | Labels |
|---|---|---|---|
emco_deployment_intent_group_resource | GAUGE | 0 or 1 | project |
The deployment intent group shown in Example query - status=deployed would create the following metrics:
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="ConfigMap.v1",resource="firewall-scripts-configmap",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="Deployment.v1.apps",resource="fw0-firewall",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge02",connectivity="available",resource_gvk="Config.v1",resource="firewall-scripts-configmap",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge02",connectivity="available",resource_gvk="Deployment.v1.apps",resource="fw0-firewall",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="packetgen",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="Deployment.v1.apps",resource="fw0-packetgen",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="packetgen",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="ConfigMap.v1.apps",resource="packetgen-scripts-configmap",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="packetgen",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="Service.v1.apps",resource="packetgen-service",resource_deployed_status="applied",resource_ready_status="ready"}
...
Some example queries:
Description | PromQL |
|---|---|
deployedCounts | count(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",resource_deployed_status="applied"}) |
readyCounts | count(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",resource_ready_status="ready"}) |
count(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",resource_ready_status="notready"}) | |
apps | count(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group"}) by (app) |
clusters filtered by the sink and firewall apps | count(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",app="sink"} or emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",app="firewall"}) by (cluster_provider,cluster) |
Tracing
Istio provides a starting point for tracing by creating a trace for each request in the sidecars. But this is insufficient as it does not include the outgoing requests made during an inbound request. What we'd like to see is a complete trace of, for example, an instantiate request to the orchestrator that includes the requests made to any controllers, etc.
In order to do this it is necessary to pass the tracing headers from the inbound request through to any outbound requests. This will be done with the https://opentelemetry.io/ golang libraries.
Logging
Each log message must contain the timestamp and identifying information describing the resource, such as project, composite application, etc. in case of orchestration.
The priority is placed on error logs; logging other significant actions is secondary.