This is a work in progress to define the minimum viable/valuable product (MVP) for a program to validate the life-cycle of a VNF running "in/with" ONAP.

The MVP definition should not discourage anyone from contributing to other projects or efforts related to VNF testing, but should help guide developer priorities for test frameworks.

Requirements & Deliverables 

1. The goal of the MVP is to specify a suite of VNF validation tests and associated test infrastructure that can be developed within the time-frame of the E-release of ONAP.

2. Tests will focus on on-boarding and instantiation of the VNF in/within ONAP.

3. Tests will focus on HEAT and TOSCA based VNFs.

4. Test will leverage the SDNC and APPC, VFC controllers.

5. Tests will use the the VIM that is provided with ONAP i.e. a generic version of OpenStack. (a future program update / release may migrate to other VIMs).

6. VNFs are validated against the specific release of ONAP (i.e. VNF is validated against ONAP E-release or F-release).

7. VNFs must also past the current VNF criteria defined by ONAP for the initial release of VNF compliance testing.

8. Testing uses existing interfaces into ONAP for stimulus / response to "drive" the tests (i.e. avoid creating new requirements / interfaces for the ONAP at large project).

What is needed / Work to do

1. ONAP release must be readily deploy-able, to allow a test framework to run on the deployment and test a VNF.  

2. ONAP deployment must be reproducible to ensure VNF tests are conducted in a uniform environment.

3. Definition of life cycle requirements (i.e. what the test cases validate)

   1. Both HEAT & TOSCA 

      1. What does a mean (formally for a test case requirement) to "instantiate" (or startup) a VNF?

         1. 
            
            

            
            
            

         2. 
            
            

            
            
            

      2. Does instantiation include configuration?

      3.  

         1. Victor: both heat and Tosca template could include the inject-file/user-data(cloud-init) as the day0 configuration file of a VNF.  

         2. Trevor: I would argue against configuration for the MVP phase.  There are a variety of options to execution post-instantiation configuration management (Ansible, Chef, etc.).   Since there are multiple options for a compliant VNF to utilize it would not make sense to offer only a subset of options, and offering any option would involve setup, configuration, and execution of the configuration mgmt option itself. 

      4. Does instantiation include health check?

         1. Victor: Could be yes, we could use the query interface (query the VNF detail info) to implement the health-check func.

      5. HEAT

         1. @Trevor Lovett & @Ryan Hallahan will help with this.

         2. The following is a DRAFT - needs further review and input

         3. Prerequisite Decision Points 

            1. Are we planning for decentralized (VNF Provider must setup their own ONAP instance and peform their own testing) or centralized testing (VNF Provider is submitting their VNF and associated settings for testing)  model?  Assuming the former, but need confirmation as the later has additional automation and other concerns to care for (network creation, image registration, etc.)

         4. Assumptions 

            1. Test scripts will utilize a default license model in SDC

            2. Compliance with test suite assumes the VNF has already has compliant packaging certified

            3. If utilizing full automated testing, then assume the Service model will only include a single VF for instantiation.  It would likely be too difficult to automatically create a multi-VF service in the first phase, and not necessary for the scope of testing

            4. Test Input will include the preload information in some TBD file format that can be loaded into SDNC.

         5. Implementation Options 

            1. Within ONAP today, the Integration Testsuite generally has all the building blocks to automate and execute the steps below.   It would need some modification to handle a more arbitrary VNF as it is not used in this way today, but based on preliminary analysis this looks achievable. 

            2. Orange Python Framework - This is another option that can be considered.  It looks to handle the instantiation use case, and potentially others.  

         6. Potential Test Flow

            1. Intialize Vendor and Category Information

            2. Create the VSP in SDC

            3. Upload Heat Archive

            4. (Optional) Assign any Unassigned Files to Artifacts

            5. Validate the VSP and ensure now Errors exist (warnings are OK)

            6. Assign the Vendor License Model to the VSP (assumes a single VLM for testing purposes)

            7. Create the Virtual Function

               1. Import the VSP (find using Name or ID from prior steps)

               2. Set name of VF (auto-assign or make input into test script), contact and other required fields

            8. Create Service

               1. Set Name (auto-assign based on VSP or make input into test script)

               2. Assign required or optional fields based on test script input

               3. Assign VF to the Servce Model

            9. Distribute the Service Model and validate successful Distribution

            10. Submit Preloads to SDNC

            11. Trigger Instantiation of Base Module from VID (NOTE: Need to see how we handle multi-module VNFs - presumably we can query this information and instantiate each individully)

            12. Verify sucessfull instantiation

            13. Health-check TBD - needs further discussion

      6. TOSCA

         1. @victor gao will help with this.

         2. 
            
            

            
            
            

   2. Test case description template for specifying VNF validation test purpose, implementation steps and pass/fail criteria.

   3. Definition of the set of ONAP components and their configuration required for the testing (ONAP profile used for testing).

   4. Definition of test infrastructure requirements needed for testing (i.e. hardware with compute / network / storage and pod requirements to run testing).

   5. Test tooling that drives testing through existing ONAP interfaces.

      1. HEAT

      2. TOSCA

What is needed / Work to do Matrix

Test Life Cycle

Items to do 

Based on a review with the ONAP Integration team the test suite Robot scripts provide the majority of the building blocks to perform the automation required for Heat-based VNFs this effort.   There is still work to adapt the existing scripts to handle a generic VNF vs. the predefined demo VNFs currently used.  The amount of effort on a per function basis as laid out in the table is not known at this time, but the overall effort does look to be achievable in the El Alto time frame.

Open Questions

1. Can the test requirements or definitions (procedures) by pulled from, or reuse, the ETSI TST-0007

   1. What are the integration and testing interfaces that are currently available, i.e. used by the integration team / gating team?

Definition of Done / Success Measures

1. Tests can readily be run, with high level of repeatability.

   1. Level of complexity is manageable by end users (i.e. ease of ONAP deployment + test cases).

EUAG Feedback

Please place your feedback here, as needed.

1. Feedback from CMCC: The VNFD we are using in our company are all TOSCA-based. Also, we are using VFC for VNF LCM. We suggest to update the 3rd item and the 4th item in "Requirements & Deliverables" to  "Tests will focus on HEAT based VNFs and TOSCA based VNFs" and "Test will leverage the SDNC, APPC and VFC controllers".

   1. Verizon feedback: As mentioned in the comment below we we are using TOSCA based VNF-D as well. We would like to extend the requirements to include SOL004/SOL001 VNFs using the SDC→SO→SOL003 Adapter → External VNFM→VNF path or the SDC → SO → SOL005 Adapter → VF-C → VNF path.

   2. China Telecom Feedback：We have both HEAT based and TOSCA based VNFDs in the DEMO running on our testbed. It will be great if TOSCA based VNFs could be added in the Requirement, providing the resources are available.

   3. ChinaUnicom Feedback： Our company also has the strong demands for TOSCA based VNF and we hope that it could be included into the scope.

Timeline

External Resources

https://wiki.onap.org/display/DW/OVP+LCM+Support

Testing framework comparison