Overview
The purpose of this page at this time is to capture requirements related to observability of the EMCO services (https://gitlab.com/groups/project-emco/-/epics/7).
...
Also, keep in mind this cautionary note from the Prometheus project
CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.
Unbounded sets of values in the EMCO APIs would include values such as project names, intent names, etc.
Preliminary metrics
This section contains some of the considerations of the guidelines above applied to the orchestrator and rsync services.
Actions may be either HTTP or gRPC requests:
| Service | Action |
|---|---|
| orchestrator | approve |
| instantiate | |
| migrate | |
| rollback | |
| stop | |
| terminate | |
| update | |
| StatusRegister | |
| StatusDeregister | |
| rsync | InstallApp |
| UninstallApp | |
| ReadAppContext | |
| Alert | |
| Unsubscribe | |
| UpdateApp | |
| RollbackApp |
The requests, errors, and latency can be modeled after Istio's istio_requests_total and istio_request_duration_milliseconds, with an additional action name label.
Tracing
Istio provides a starting point for tracing by creating a trace for each request in the sidecars. But this is insufficient as it does not include the outgoing requests made during an inbound request. What we'd like to see is a complete trace of, for example, an instantiate request to the orchestrator that includes the requests made to any controllers, etc.
...