CNTT Field Trial Plan

Owned by Scott Steinbrueck
Last updated: Apr 07, 2020
4 min read



Order of priorities

  1. Field Trial + RC (functional tests with Field Trial participants)

  2. RI + RC (function tests within the RI lab environment)

  3. Additional RC testing (hardware validation, performance, etc.)



What is the Field Trial

  1. The first step towards CNTT Adoption

  2. Prove that the RC tests work at a basic level (functional testing)

  3. Work with key NVFi vendors (targeting Mirantis, Redhat)



How will the Field Trial work

  1. CNTT to document the process (pending PR at https://github.com/cntt-n/CNTT/tree/master/doc)

    1. For example...

      1. How will CNTT (OPNFV) operate

      2. What will the participants need to do

      3. What are the survey questions

  2. Participants to setup their environments (or use community lab), then run the functional tests

  3. Participants to complete a survey list of questions (many answers will be optional, including the test results)



What is still needed for the Field Trial

  1. Completed CNTT process document at Github

  2. Agreement with NFVi vendors

  3. Finalize RI Cookbook

    1. requirements for running in private lab

    2. requirements for running in community lab

  4. RC Cookbook

    1. Test case mapping to RA (requirements) - RC Ch 3

    2. How to run the test cases - RC Ch 7 ?

  5. Plan for test results

    1. Display results in graphical format (consider who has access to view ?)

    2. Allow vendor to submit results for badging



When will the Field Trial run

  1. Late April - into May

  2. Originally intended to coincide with ONES

  3. Pending talks with NFVi vendors



What about the existing efforts in CNTT / OPNFV

  1. RI and RC efforts will continue to move forward in parallel

    1. RI - Focus on issues such as:  Cookbook / Design / Lab setup and fixes

    2. RC - Continue enhancing test suites, getting test results to dashboard (Dovetail)

  2. People

    1. WSL-RI:  Raja Jayachandran has been nominated

    2. James Gu (existing OPNFV-Airship proxy PTL)

  3. Other types of testing can be included in the future Badging process



Next steps

  1. Raja to connect with Fu Qiao (WSL RI + RC alignment)

  2. Raja to work through RI - Issues and PRs

  3. Raja and James will work together to clarify possible issues in community areas Airship, Openstack Helm, Treasuremap and others

  4. Work through section "What is still needed for the Field Trial"

  5. What resources are needed to help towards the Field Trial priority



Detailed Plan / Close Gaps

  1. Close gaps

  2. Alignment plan between Field Trial team, RI, RC, OPNFV

  3. Staffing needs

Legends: Green resources are needed for the completion of the trial on Time.

High Level Item

Tasks

Staffing needs for Trial

Notes

CNTT

OPNFV



  1. CNTT process 











Document the process in GitHub

MMA: Scott Steele (CNTT)





2. Agreement with NFVI vendors











Reach out to NFVI Vendors.

MMA: Bob and Scot Steele (CNTT)





3. Finalize RI Cookbook











a. Document Requirements for running in private lab (e.g. Min hardware specs needed for VI).

RI-WS: 







b. Document Requirements for running in community lab.

RI-WS:



Community labs will not be available for VI to be installed.



c. Lab: Getting the labs ready in place. (Labs)



TBN: TBA

Community labs will not be available for VI to be installed.



d. Installer 



Airship: TBA

NOT Needed for Field trials.

4. RC Cookbook











a. Document Test case mapping to RA (requirements) 

RC1-WS: (Ch03)



Cleaning up might be needed.



b. How to run the test cases - RC Ch 7 ?

RC1-WS: (Ch07)



Cookbook in place.



c. X-testing (tool)



Xtesting: 1 resource.

Maintain and support testing candidates that comes in board.



d. X-testing → Dovetail integration



Xtesting: TBC

Cedirc will confirm if resources are needed for Xtesting.



d. X-testing: Functional (Test Scripts)



Functest: 1-3 resources

More Help need to enrich test cases and cover all functional requirements in CNTT.



e. X-testing: Performance (Test Scripts)



StorPerf: 1 resource (allowed to work other X-testing issues)

vSperf: TBA

Prox: TBA

NFV Bench:  TBA

@Al Morton to confirm number of resources needed per each project.

TBA resources needed post-Field Trial



f. X-testing: High Availability (Test Scripts)



New: TBA



5. Test results











a. Portal and Displaying results and submit result in. Display results in graphical format. Allow vendor to submit results for badging (consider who has access to view ?)



Dovetail: 1 resource.

To maintain dovetail web portal and make sure format remains compatible with X-testing interfaces



b. Collect feedback to CNTT to enhance RC.

MMA: 







c. Test API and Testdb need enhancement to provide for complete test execution records, as well as CNTT requirements mapping and on-going curation.



???: 1 resource.

Not clear what project is maintaining the Test API and Testdb.



  1. Al - OPNFV has done this before, through badging process.

    1. Vendor feedback @ ONS 2018 - how to include performance, because their customers are asking for this. 

    2. Also what about High Availability (HA). 

    3. Security scenarios need proper attention.

  2. Rabi - CNTT has defined a single OSTK-based configuration, which helps limit the scenarios. 

    1. The approach taken provides the framework in which additional testing can be added.

    2. WSL RC can help drive the right focus in the RC test plan.