Testing Evolution & Strategy
- Lincoln Lavoie
- Trevor Cooper
- Rabi Abdel
- Margaret Chiosi
Discussion and opens:
conformance vs compliance.
Standard vs Specs.
?
Note, the following is a work in progress.
Overview
This page is intended to capture LFN CVP committee discussions and work relating to the longer term strategy and direction of testing against industry standards, specifications and references within the LFN projects. LFN CVP is responsible for strategy, process and oversight of compliance and verification programs within LFN. Compliance programs have commercial involvement and hence require strong governance including legal considerations. Technical oversight for compliance programs (test tools, test cases, test criteria, etc.) are the responsibility of designated TSC (currently OPNFV for NFVI and ONAP for VNF).
Definitions
Standard (Industry Standard): A specification published by an industry standards body (could be draft or proposed).
Specification: Technical norm or reference agreed by an industry group (e.g. open source community) which is not a standard.
Compliance: A "device" is compliant if it meets the requirements defined within a specification, i.e. a power plug is required to have 3 prongs and would be consider compliant if it indeed has 3 prongs in the correct locations and orientations.
Conformance: ???
Certification: ???
Validation: A "device" is validated when it operated according to its intended purpose or design, i.e. a power plug is able to connect to a socket that is also known to be compliant and passes electricity, both have been validated (it really works).
Test domain:
Functional
Performance
Security
Compliance
VNF Compliance
Conformance to HEAT, TOSCA, or HELM Templates, as defined by ONAP Requirements.
Conformance to ONAP VNFRqmts documentation which will integrate other SDO requirements (ie, 5G, BBF, ETSI,...)
Conformance to ONAP interfaces.
Documentation of VNF is available.
NFVI Compliance:
Compliant against an NFVI reference platform with specific capabilities (e.g. functional, performance, security, etc.).
Conformance may include interface standards e.g. ETSI ISG NFV (or a subset of those requirements) as well as well accepted API specifications and operation (i.e. OpenStack or K8s APIs)
Open Questions:
Should the compliance tests apply to both software and hardware?
Hardware example: NUMA compliance, DPDK support, ODP compliance
Support DPDK and/or FDIO
Address additional specification (e.g. GSMA profiles)
We need a standard to comply against. What is it? GSMA profile.
Validation
VNF Validation:
On-boarding: Validate the VNF Artifact(s) is able to be "consumed" by an ONAP instance.
Open questions: 1) What are the requirements defined for "On-boarding?" What is the requirement for the "ONAP instance?" 3) Is there a requirement for backwards compatibility? 4) Besides ONAP are there other organizations defining this?
Open question: 5) ONAP instance: does it mean MANO, or does it include VNFM.
Instantiation: Validate the VNF can be "launched" by an ONAP instance on top of an NFVI.
Open questions: 1) What is the requirements for ONAP Instance? 2) What is the requirements for the NFVI? 3) What are the requirements for the "VNF launch?"
Validate various life-cycle operations are possible with the VNF running on the ONAP Instance.
Open questions: 1) What the required "life-cycle operations?" 2) What the requirements for each "operation?" 3) What are the requirements for the "ONAP instance?"
Life Cycle Operations of VNF (long term vision) - my version of FCAPS:
Fault Management
Capacity Management
Billing/Accounting Management
Performance Management
Security Management
Open question: does it relate to MANO, or does it include VNFM (generic vs specific).
NFVI Validation:
Instantiate NFVI/VIM
Life Cycle of NFVI/VIM - Healthcheck?
Open Questions:
ETSI definition of NFVI
Performance
NFVI Performance
Capacity: of RAM, CPU, storage
Stability:
Chaos monkey tests?
data throughput, CPU performance, RAM/Storage access performance?
What is my reference implementation of NFVI? GSMA Profiles (B, N, and C). ETSI ISG NFV has a performance working group having specs. Led by Al Morton, AT&T. Also I think Pierre Lynch, IXIA, leads the testing spec group also.
What are the metrics we need to measure? (align with ETSI, GSMA defined metrics).
1 test per metric?
VNF Performance
huge dependency on NFVI. need reference NFVI to compare against. (GSMA Profiles).
what are the metrics we need to measure? (align with ETSI, GSMA defined metrics).
1 test per metric?
VNF Reference Implementation.
Security
Open question: Where do these requirements come from? Who is responsible for the test cases?