OVP Phase 2 Project Mapping and Relationships
Status: DRAFT
Overview
This is an attempt to map the ecosystem Open Source projects in the OVP Phase 2 umbrella and their relationships. The goal is to link the source of “what to validate” (i.e. requirements) to either testing specifications (i.e. Certifications) or testing implementations. A project being in a box only implies it is potentially aligned to provide the functionality – not that it already has it or we confirmed alignment.
Additionally, we are analyzing areas where there is ambiguity or lack of clarity, and recommending alternative approaches to either eliminate the issue or make the scope clearer.
Mapping (Current Categories)
Category | Sub Category | Requirements | Conformance | Test Impl./Tools | Notes |
Cloud Platform Conformance | Performance/NF | ??? | ??? | ??? | Doesn’t appear to be in RA-2 |
| Functional | RA-2 | RC-2 | ??? | Are there tools to specifically validate RA-2? |
| Cloud Native | RA-2 | RC-2 | CNCF Software Conformance (K8S compatibility only) | CNF Conformance has defined limited tests against the K8S instance itself (CNI and K8S APIs) |
CNF On-boarding | CNF Compliance | CNF Conformance ONAP VNF Requirements | RC-2 | CNF Conformance | CNF Conformance is not specific to CNTT nor part of LFN ONAP CNF support is limited and ONAP is not currently a requirement for a CNF to utilize an RI based on RA-2. |
| CNF LCM | CNF Conformance | RC-2 | CNF Conformance |
|
CNF Conformance | Performance | ??? | ??? | ??? | Should this all include Non-Functional as we did with Cloud Platform Conformance? Unclear what we are testing against here. |
| Functional | ??? | RC-2 | ??? | Need better definition of what this would encompass. Are we checking that a NF does the function (ex: firewall) that it supposed to perform? |
| Cloud Native | CNF Conformance | RC-2 | CNF Conformance CIS Benchmark |
|
General Questions/Comments/Concerns on Current Categorization
Cloud Platform Performance - Do we envision that there will be specific requirements related to the performance of NFVI expressed as some minimum level of performance? UIf so where do we envision such expectations being documented? Alternatively, do we see this primarily as a benchmarking exercise where reference CNFs and tests will be run to produce results? If so, is this really in the scope of OVP or should benchmarking be a separate activity?
CNF Onboarding and CNF Conformance don't appear to be separate concerns. They are still functions of the CNF's compliance with CNTT and general Cloud Native/K8S requirements. Given the current architecture of RA-2, there is also not a separate onboarding activity that is separate from instantiation - this is a difference from the VNF badge with ONAP. Recommend collapsing these to a single category.
Functional vs. Cloud Native - This distinction is not clear, and there are likely to be overlaps between the 2 categories. Working with the assumption that CNTT will be the source of requirements, if there is going to be a breakdown it must be very clear what the distinction is or align to a breakdown in the source requirements (i.e. RA-2). Recommend collapsing these categories.
CNF Performance - This type of testing is often very specific to the function of the CNF. We're not clear that a general purpose framework could be created nor requirements could be defined. We recommend at a minimum this is outside the MVP scope, and it may be outside of OVP scope as well.
Other Open Questions/Concerns
No project in the LFN umbrella is currently defining requirements for what a CNF needs to comply with. The only project identified is the CNF Conformance project which is part of CNCF TUG. Given that a CNF will need to be compatible with a given CNTT RA and RI, it would seem there needs to be a project somewhere in CNTT to manage those requirements and potentially tests. If there are more general Cloud Native requirements that aren't specific to the CNTT, then we could create a requirement to comply with those external requirements and test suites.
There was mention of using ONAP as a vehicle to define CNF onboarding requirements, but this presents some challenges:
ONAP’s support of CNFs is fledgling and still under development. As such, no requirements currently exist for CNFs in ONAP and they are likely go under significant change as support is built out.
Additionally, ONAP is not required for a CNF to utilize an RA-2 based implementation.
Suggestion: We need a place within CNTT to define the requirements of a CNF to be compatible with RA-2. ONAP may define additional requirements to onboard a CNF into ONAP for orchestration and management, but that would be a separate and optional set of requirements that would only be applicable when using ONAP. This may impact how the badges are defined as well.
RC-2 does not plan to handle certification of the CNFs, but rather the NFVIs. However, the diagram on the OVP 2.0 Boot Strap page implies that RC-2 does play a role. We either need to change the scope of RC-2 or correct the diagram.
Answer: Per discussion with RC-2 lead, we agreed that some level of CNF requirements can be included in RC-2 (this would someone align with the approach for RC-1). However, unlike RC-1 there are no external CNF-specific requirements within LFN to point to. In RC-1, they were pointed to ONAP testing and requirements specific to the VNF.
Suggested Revised Categorization
Cloud Platform Compliance
Performance/Non-Functional - Does the NFVI meet specific performance benchmarks using reference CNFs , and does it handle specific non-functional requirements
Note: This assumes that we envision there will be specific performance requirements. If this is not true, then this category should be revisited.
Functional - Does the NVFI support the required capabilities as defined by RA-2
Note: If further sub-categorization is required, then it should be tied to categorization of requirements in CNTT
CNF Conformance
Artifact Compliance (images, descriptors, charts, etc.) - Does the CNFs charts, configurations, descriptors, images, etc. meet CNF requirements as defined in CNTT?
Functional - Can the CNF perform specific general purpose LCM activities on a RA-2 compliant NFVI.
Note 1: This is not testing the the Network Function behavior itself (i.e. it operates properly as a firewall) which is out of scope of OVP
Note 2: If further sub-categorization is required (security, , then it should be tied to categorization of requirements in CNTT
Mapping (Proposed Categories)
Category | Sub Category | Requirements | Conformance | Test Impl./Tools | Notes |
Cloud Platform Conformance | Performance/Non-Functional | ??? | ??? | ??? | Where are non-functional requirements documented? Are requirements for performance expected? Should benchmarking against ref. CNFs be it's own track? |
| Functional | RA-2 | RC-2 | CNCF Software Conformance (K8S compatibility only) | Current tools are not currently linked to RA-2, but CNF Conformance is analyzing alignment. Are there other tools/projects that will test specific requirements? CNF Conformance primary focus so far has been on testing the CNF itself, although I see some tests specific to the NFVI. Do we see this suite testing both NFVI and CNFs? |
CNF Conformance | Artifact Compliance (images, descriptors, charts, etc.) | RC-2 | We would still need additional requirements specific to RA-2, and potentially more general purpose telco requirements. Where would those be documented? RA-2, RC-2, or somewhere else? | ||
| Functional | ??? | RC-2 | ??? | This testing would not cover the functional behavior of the CNF (e.g. is it a firewall), but rather can the CNF handle standard LCM operations or utilize capabilities of RA-2 based NFVI properly. There is no place in CNTT or any LFN project where such requirements are defined for CNFs |
Comments
Thanks, Bill. Re: the scope of RC-2, I've updated the mapping table to increase the scope of RC-2. Please review and let me know if that is a better reflection. I did review RC-1, and it does have sections dedicated specifically to VNF verification/validation, so I assume that will be carried over to RC-2. The actual requirements of the tests performed are very high-level, and ultimately refer back to ONAP requirements for the bulk of the heavy lifting. So while RC-2 will cover CNF testing, I think we'll still have to solve for what CNF requirements are we going to validate and where those are defined.
Re: the CNF Conformance tests, they are individual, but I think we will have to ensure the individual tests are organized into a testing profile for easy selection and execution. That could be separate projects or simply separate testing profiles that are selected at run time.
I've only spent a little time looking at CNF testbed. If you're more familiar with it, maybe you could speak a bit about it on our next working session so we understand how it fits into the picture.
As for the categorizations, I'm open to other ways to do it. What I laid out was mostly a straw-man proposal aimed at simplifying the current structure. My hope is we can align on a proposal as a work stream, and take it back to the larger group. I'd like to dedicate some time to this on our next meeting, then potentially bring it to the next CVC call.
For the Cloud Platform Compliance, I think the Performance/Non-Functional category lies outside the scope of CNTT and OVP. Performance seems to be more between the vendor and operator and I think it would get hard for any of them to actually publish numbers even if we came up with tests.
For CNF Conformance, if we want to define conformance, there is currently a massive gap in RA2 because it is not addressed there at all. The conformance can be done in RC2, but the requirements need to come from somewhere so this is a gap that we need to bring back to RA2.
I would tend to agree with you on Performance and Non-Functional. Let's discuss on our weekly call how to address this.
For CNF Conformance, I had a typo in the table, CNF Functional requirements should have been listed as an open instead of listing RA-2. I've updated it.
Open Questions/Issues
RC-2 does not plan to handle certification of the CNFs, but rather the NFVIs. However, the diagram on the OVP 2.0 Boot Strap page implies that RC-2 does play a role. We either need to change the scope of RC-2 or correct the diagram.
From Rabi - Although RC2 will mainly focus on the NFVI, Conformance of CNFs running on top of the NFVI is also covered. this will however look at how CNFs interact with the infrastructure (using the standard defined interfaces). Although the scope of the RC2 will mainly be infrastructure, it will also cover CNF conformance as well (to be limited to how CNFs are consuming infra resources). so it might make sense to clarify this to the RC2 Scope
CNF Conformance currently has tests for CNFs and the Kubernetes platform itself (ex: CNI tests, K8S API tests). CNF Providers and NFVI providers are different use cases. Should we split these tests?
Aren't these all individual tests right now anyways so they can be split up and recombined as needed?
No project in the LFN umbrella is currently defining requirements for what a CNF needs to comply with. The only project identified is the CNF Conformance project which is part of CNCF TUG. Given that a CNF will need to be compatible with a given CNTT RA and RI, it would seem there needs to be a project somewhere in CNTT to manage those requirements and potentially tests. If there are more general "Cloud Native" requirements that aren't specific to the CNTT, then we could create a requireement to comply with those external requirements and test suites.
I agree and think this should be brought back to the larger group. Should this be part of LNF or does it lie outside its scope?
There was mention of using ONAP as a vehicle to define CNF onboarding requirements, but this presents some challenges:
ONAP’s support of CNFs is fledgling and still under development. As such, no requirements currently exist for CNFs in ONAP and they are likely go under significant change as support is built out.
Additionally, ONAP is not required for a CNF to utilize an RA-2 based implementation.
Suggestion: We need a place within CNTT to define the requirements of a CNF to be compatible with RA-2. ONAP may define additional requirements to onboard a CNF into ONAP for orchestration and management, but that would be a separate and optional set of requirements that would only be applicable when using ONAP. This may impact how the badges are defined as well.
See the above response from Rabi. It seems RA2 could define how CNF should be compatible with the infrastructure while RC2 could be used for verification. This approach seems similar to CNF testbed that tests both infrastructure and NFs.
Testing Category Feedback
The 2 categories of CNF Conformance and CNF Onboarding seem to significantly overlap. Onboarding artifacts would seem to be just another aspect of CNF Conformance rather than a category itself.
I agree. Can they be combined into one category?
In Cloud Platform Compliance, the distinction between the categories Cloud Native and Functional are not clear to me. Is a distinction necessary? RA-2 already has requirements such as the implementation must be compatible with K8S software conformance tests. If we are going to have sub categories of compliance, then I suggest we better align them with RA-2 categories (ex: Capabilities, Architecture, etc.)
I agree. I don't think we actually need sub categories for cloud native and functional. However, I do think performance should be left as its own category.
Under CNF Conformance, it's unclear what Functional means. Does it mean the CNF performs the function it is designed to do (e.g. firewall) or does it mean it is compatible with specific lifecycle functions common to all CNFs
I think it is the latter the it functions like a CNF should. With this in mind then this is the same as for infrastructure above that functional and cloud native should just be combined into one category.
Suggested Categorization:
Cloud Platform Compliance
Performance/Non-Functional - Does the NFVI meet specific performance benchmarks, and handle specific non-functional requirements
RA-2 Capabilities Alignment - Does the NFVI include all mandatory capabilities defined in RA-2
RA-2 Architecture Alignment - Does the NFVI meet the mandatory architecture requirements defined in RA-2
Do capabilities and architecture need to be separate? See above
CNF Conformance
Artifact Compliance (images, descriptors, charts, etc.) - Do the charts, configurations, descriptors, images, etc. meet CNF requirements
Functional - need a better definition of the scope of this
Performance - not sure what would be validated here. It wouldn't seem to be a matter of conformance. Maybe CNF Benchmarking is something that should be in it's own category.
TO DO:
Clarify RC2 Scope regarding CNFs