This is a work in progress to define the minimum viable/valuable product (MVP) for a program to validate the life-cycle of a VNF running "in/with" ONAP.

The MVP definition should not discourage anyone from contributing to other projects or efforts related to VNF testing, but should help guide developer priorities for test frameworks.

Requirements & Deliverables 

1. The goal of the MVP is to specify a suite of VNF validation tests and associated test infrastructure that can be developed within the time-frame of the E-release of ONAP.
2. Tests will focus on on-boarding and instantiation of the VNF in/within ONAP.
3. Tests will focus on HEAT and TOSCA based VNFs.
4. Test will leverage the SDNC and APPC, VFC controllers.
5. Tests will use the the VIM that is provided with ONAP i.e. a generic version of OpenStack. (a future program update / release may migrate to other VIMs).
6. VNFs are validated against the specific release of ONAP (i.e. VNF is validated against ONAP E-release or F-release).
7. VNFs must also past the current VNF criteria defined by ONAP for the initial release of VNF compliance testing.
8. Testing uses existing interfaces into ONAP for stimulus / response to "drive" the tests (i.e. avoid creating new requirements / interfaces for the ONAP at large project).

What is needed / Work to do

1. ONAP release must be readily deploy-able, to allow a test framework to run on the deployment and test a VNF.  
2. ONAP deployment must be reproducible to ensure VNF tests are conducted in a uniform environment.
3. Definition of life cycle requirements (i.e. what the test cases validate)
   1. Both HEAT & TOSCA 

      1. What does a mean (formally for a test case requirement) to "instantiate" (or startup) a VNF?

         1. from Victor, Definition of VNF onboarding for TOSCA based VNF.

            • Verify the VNF package by using VNFSDK compliance check test cases.
            • upload a VNF package to ONAP system
            • verify the VNF Package already exist on the ONAP system.

         2. from Victor, Definition of VNF instantiation for TOSCA based VNF.

            • Trigger an "instantiate VNF" operation to Controller
            • verify that the requested grant for the "instantiate VNF" operation has been approved by the controller
            • VNF related software images have been successfully added to the image repository managed by the VIM or already exist on the VIM system(Currently, ONAP is manually doing this.)
            • Verify that the requested virtualised resources have been allocated by the VIM according to the VNFD
            • Verify that virtualised resource allocation constraints have been met by querying the VIM
            • Verify that any existing virtualised resources have not been affected by the allocation of the new virtualised resources by querying the VIM
            • Verify that the VNF instance resources are visible on the controller
      2. Does instantiation include configuration?
      1. Victor: both heat and Tosca template could include the inject-file/user-data(cloud-init) as the day0 configuration file of a VNF.  
      2. Trevor: I would argue against configuration for the MVP phase.  There are a variety of options to execution post-instantiation configuration management (Ansible, Chef, etc.).   Since there are multiple options for a compliant VNF to utilize it would not make sense to offer only a subset of options, and offering any option would involve setup, configuration, and execution of the configuration mgmt option itself. 
      3. Does instantiation include health check?
         1. Victor: Could be yes, we could use the query interface (query the VNF detail info) to implement the health-check func.
      4. HEAT
         1. Trevor Lovett & Ryan Hallahan will help with this.
         2. The following is a DRAFT - needs further review and input
         3. Prerequisite Decision Points 
            1. Are we planning for decentralized (VNF Provider must setup their own ONAP instance and peform their own testing) or centralized testing (VNF Provider is submitting their VNF and associated settings for testing)  model?  Assuming the former, but need confirmation as the later has additional automation and other concerns to care for (network creation, image registration, etc.)
         4. Assumptions 
            1. Test scripts will utilize a default license model in SDC
            2. Compliance with test suite assumes the VNF has already has compliant packaging certified
            3. If utilizing full automated testing, then assume the Service model will only include a single VF for instantiation.  It would likely be too difficult to automatically create a multi-VF service in the first phase, and not necessary for the scope of testing
            4. Test Input will include the preload information in some TBD file format that can be loaded into SDNC.
         5. Implementation Options 
            1. Within ONAP today, the Integration Testsuite generally has all the building blocks to automate and execute the steps below.   It would need some modification to handle a more arbitrary VNF as it is not used in this way today, but based on preliminary analysis this looks achievable. 
            2. Orange Python Framework - This is another option that can be considered.  It looks to handle the instantiation use case, and potentially others.  
         6. Potential Test Flow
            1. Intialize Vendor and Category Information
            2. Create the VSP in SDC
            3. Upload Heat Archive
            4. (Optional) Assign any Unassigned Files to Artifacts
            5. Validate the VSP and ensure now Errors exist (warnings are OK)
            6. Assign the Vendor License Model to the VSP (assumes a single VLM for testing purposes)
            7. Create the Virtual Function
               1. Import the VSP (find using Name or ID from prior steps)
               2. Set name of VF (auto-assign or make input into test script), contact and other required fields
            8. Create Service
               1. Set Name (auto-assign based on VSP or make input into test script)
               2. Assign required or optional fields based on test script input
               3. Assign VF to the Servce Model
            9. Distribute the Service Model and validate successful Distribution
            10. Submit Preloads to SDNC
            11. Trigger Instantiation of Base Module from VID (NOTE: Need to see how we handle multi-module VNFs - presumably we can query this information and instantiate each individully)
            12. Verify sucessfull instantiation
            13. Health-check TBD - needs further discussion
      5. TOSCA
         1. victor gao will help with this.

         2. from Victor, Definition of VNF instantiation for TOSCA based VNF.

            • Trigger an "instantiate VNF" operation to Controller
            • verify that the requested grant for the "instantiate VNF" operation has been approved by the controller
            • VNF related software images have been successfully added to the image repository managed by the VIM or already exist on the VIM system(Currently, ONAP is manually doing this.)
            • Verify that the requested virtualised resources have been allocated by the VIM according to the VNFD
            • Verify that virtualised resource allocation constraints have been met by querying the VIM
            • Verify that any existing virtualised resources have not been affected by the allocation of the new virtualised resources by querying the VIM
            • Verify that the VNF instance resources are visible on the controller
   2. Test case description template for specifying VNF validation test purpose, implementation steps and pass/fail criteria.
   3. Definition of the set of ONAP components and their configuration required for the testing (ONAP profile used for testing).
   4. Definition of test infrastructure requirements needed for testing (i.e. hardware with compute / network / storage and pod requirements to run testing).
   5. Test tooling that drives testing through existing ONAP interfaces.
      1. HEAT
      2. TOSCA

What is needed / Work to do Matrix

Test Life Cycle

Items to do 

Based on a review with the ONAP Integration team the test suite Robot scripts provide the majority of the building blocks to perform the automation required for Heat-based VNFs this effort.   There is still work to adapt the existing scripts to handle a generic VNF vs. the predefined demo VNFs currently used.  The amount of effort on a per function basis as laid out in the table is not known at this time, but the overall effort does look to be achievable in the El Alto time frame.

Open Questions

1. Can the test requirements or definitions (procedures) by pulled from, or reuse, the ETSI TST-0007
   1. What are the integration and testing interfaces that are currently available, i.e. used by the integration team / gating team?

Definition of Done / Success Measures

1. Tests can readily be run, with high level of repeatability.
   1. Level of complexity is manageable by end users (i.e. ease of ONAP deployment + test cases).

EUAG Feedback

Please place your feedback here, as needed.

1. Feedback from CMCC: The VNFD we are using in our company are all TOSCA-based. Also, we are using VFC for VNF LCM. We suggest to update the 3rd item and the 4th item in "Requirements & Deliverables" to  "Tests will focus on HEAT based VNFs and TOSCA based VNFs" and "Test will leverage the SDNC, APPC and VFC controllers".
   1. Verizon feedback: As mentioned in the comment below we we are using TOSCA based VNF-D as well. We would like to extend the requirements to include SOL004/SOL001 VNFs using the SDC→SO→SOL003 Adapter → External VNFM→VNF path or the SDC → SO → SOL005 Adapter → VF-C → VNF path.
   2. China Telecom Feedback：We have both HEAT based and TOSCA based VNFDs in the DEMO running on our testbed. It will be great if TOSCA based VNFs could be added in the Requirement, providing the resources are available.
   3. ChinaUnicom Feedback： Our company also has the strong demands for TOSCA based VNF and we hope that it could be included into the scope.

Timeline

External Resources

https://wiki.onap.org/display/DW/OVP+LCM+Support

Testing framework comparison