Minutes

Minutes of June 23 7am EDT session "Joint Topic: Performance"

• CNTT_vF2F_June_Performance-edits TCAMv02.pptx
• Work plan - use the next CVC calls (July 6) to begin documenting the agreed terms for performance. Then map how these terms apply within the context of the CNTT RC definitions.

Definitions of Performance

(SUT = System Under Test)

Calibration:

The process of checking and/or adjusting a stimulus generation or measurement device with a known reference value, to improve the overall quality of the measured results.  Calibration may be very simple, such as a comparison of the configured traffic generator sending rate and measured rate using a simple SUT (loop-back cable between interfaces), such that that the known reference value is the published nominal interface rate.

Known Reference Value:

A measured or established result or outcome for comparison with new measurements. For example, the reference value or expected outcome of a Functional Test is "PASS". The reference value or expected outcome of a Performance Measurement or Benchmarking test may be the value measured for the previous SUT release, or the published value or theoretical limit of a simple SUT.

Functional Testing:

The main object of functional testing is the verification of a specific stimulus / response behavior within the SUT, including causation.  These tests generally result in a binary outcome, i.e. pass / fail.  For example, verification of an "API call" and its associated response, such as the instantiation of a VM (or container) and verification of the VM's (or container's) existence (causation), or the ability to activate a specific feature of the SUT (i.e. SR-IOV).

Performance Measurement: 

The procedure or set of operations having the objective of determining a Measured Value or Measurement Result of an infrastructure in operation according to a defined metric.

In the context of telemetry, Performance Measurements reflect data generated and collected within the cloud infrastructure, that reflects a performance aspect of the cloud infrastructure. For example, a count of frames or packets traversing an interface per unit of time, memory usage information, other resource usage and availability, etc. This data may be instantaneous or accumulated, and made available (i.e. exposed) based on permissions and contexts (e.g., workload vs. infra).

Other Performance Measurements are designed to assess the efficiency of SUT Functions, such as the time to successfully instantiate one or more VMs or containers, or the percentage of instantiation Function failures in a set of many attempts.

Still other Performance Measurements are conducted under controlled conditions using Calibrated test systems, such that the measured results are more likely to comparable with other such measurements.

Performance Testing:

The main objective of performance testing is to understand if the System Under Test SUT is able to achieve the expected performance, through conducting a series of Performance Measurements, performance measurements and comparing those results against a specific Reference reference value.

It requires a set of performance testing tools (open source) that help with the dimensioning of a solution by an operator.

Testing results Operators need to do performance testing to help dimension a solution or to assess that a platform (particular hardware + software combination) is configured correctly and performing as expected (as compared with capacity / performance claims made by the infrastructure and VNF/CNF vendors).

Performance Testing may be useful to compare infrastructure capabilites between the System Under Test (SUT) capabilities between a particular SUT and a CNTT reference implementation of RA-2with well understood known good configurations and previously established performance ranges. Performance testing for the purpose of comparing between different commercial implementations is not a goal of CNTT and hence out of scope for the purposes of this definition. Performance testing relies on well established benchmark specifications to help establish appropriate methodologies and accuracy tolerances.

Benchmarking:

Benchmarking tests assess a key aspect of the computing environment in its role as the infrastructure for (cloud-native) network functions, using Calibrated calibrated test systems and controlled conditions. The benchmarks (and related metrics) have been agreed by the Industry and documented in publications of an accredited standards body. As a result, benchmarks are a sub-set of all possible performance metrics; they tests and metrics i.e. they are selected measurements which are more important than others. Examples benchmarks include data rate, latency, and loss ratio of various components of the environment, expressed in quantitative units to allow direct comparison between different systems treated as a black box (vendor-independence).

Because the demands on a particular system may vary from deployment to deployment, Benchmarking benchmarking assessments do not define acceptance criteria or numerical performance requirements.

Benchmarking assessment methods do not define acceptance criteria or numerical performance requirement.

Benchmark testing and Conformance conformance testing intersect when a specific requirement in the software reference architecture specification is very important , such as a frequently-executed functionto the performance of the system. Correct execution of the performance test with the expected valid result constitutes conformance. The completion time for a single conforming execution, or the number of conforming executions per second are potential Benchmark benchmark metrics, and sources of known reference values.

Bin list of related terms / topics

• Threshold - 
• Functional Tests
• Measurement Performance of functions
• Exception Threshold 
• Network
• Computation
• • CPU
  • GPU
• Life cycle
  • Time to create VM / container
  • Time to delete VM / container
  • Maximum number of VMs / containers
• SUT = System Under Test

Related CNTT issues / pull requests

[RC2 Ch01] Terminology definition for testing per issue #1495 #1649 https://github.com/cntt-n/CNTT/pull/1649

• Once terminology agreed will update this pull request to align