DRAFT (Table to be completed)

		Testsuite	VTP	Onap-tests	...

		Testsuite	VTP	Onap-tests
Project	repository	https://git.onap.org/testsuite/	https://git.onap.org/vnfsdk/refrepo/	https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests
	Leader	ONAP Integration team (Brian, Yang, Gary, Marco, ..)	VNFSDK (Victor, Kanag)	Orange
	Stats (February - may 2019)	27 contributors / 95 patches / more than 5 different companies)	3 contributors / 20 patches / 2 companies	9 contributors / 145 patches/ 3 companies
	Language	RobotFramework, Python	Java. Also supports to run test cases implemented in different languages	Python
	Links	https://wiki.onap.org/display/DW/Robot+Framework+Development+Guide https://wiki.onap.org/display/DW/Robot+Healthcheck+Tests+on+ONAP+Components	https://wiki.onap.org/pages/viewpage.action?pageId=43386304	https://wiki.onap.org/pages/viewpage.action?pageId=6593670&preview=%2F6593670%2F54722733%2Fonap_tests.pdf
Short Description		Robotframework testsuites to perform all types of tests: healthcheck, CSIT, VNF, web interface	Provides platform where VNFSDK, VVP communities test cases and others could be on-boarded and make them available over REST API, CLI and those test cases could be of different nature like compalince, life-cycle, performance and different run-time like java, python, etc. It provides test case managment, execution and result management, profile and vnf managment. Already Having integration with ONAP SDC, OPNFV dovetail, 3rd party registry. Provides capability to make use of ONAP command to consume directly as test cases over gPRC, which helps to run LCM with zero dev effort. More details LCM: https://wiki.onap.org/display/DW/OVP+LCM+Support	Python SDK for ONAP to manage VNF Life cycle based on VVP compliant Heat descriptor.
Current Usage in ONAP		release daily tests Dublin use case support including LCM tests Component CSIT test suites OOM Gating (Robot healthcheck + model distribution)	Tosca Compliance Check (OVP) VSP Compliance check (SDC design time, while on-boarding) End-End service provisioning for both HEAT and TOSCA (https://github.com/onap/integration/tree/master/test/hpa_automation )	OOM Gating (LCM tests) Third party VNF onboarding/instantiation Python CLI
VNF use cases currently supported		vFW vFWNG vFWCL vCPE vLB PNF registration ...	hpa Compalince testing	basic_vm (Ubuntu) for ONAP E2E sanity check on agting freeradius_nbi (CSIT NBI) clearwater_ims in theory any VNF with VVP compliant heat descriptor (generic framework, need config file but no additional code)
Integrations	Other ONAP components	natively integrated in onap namespace (robot pod)	Yes integrated with SDC	Black box approach versus ONAP - separation between SUT and the test tool, no hardcoded values in ONAP configurations embedded in xtesting framework (used also for infrastructure tests - same DB/API used by Dovetail for infra verification) Infra results: http://testresults.opnfv.org/test/api/v1/results?case_name=tempest_full&last=1 vnf results with ONAP: http://testresults.opnfv.org/onap/api/v1/results?case_name=basic_vm&last=1
	VVP		Yes	Yes consider VVP for heat template verification on code linting as a linter on patch submission (check all the compliancy of all refrenced heat files) based on ONAP official VVP docker
	Dovetail/OVP		Yes	No not natively but reuse all the same building block used by Dovetail (DB, test API, dockerization)
Item to do based on MVP page	Automation Script(s) to on-board VSP	Yes	Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation Script(s) to Create VF	Yes	Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation Script(s) to Create Service	Yes	Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation Script(s) to Pre-Load Image	Yes	Yes NOT an ONAP operations, and is NFVI action, will use dovetail test case	Yes
	Automation Script(s) to Submit Preloads	Yes	Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	yes
	Automation Script(s) to Instantiate VNF	Yes	Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation Script(s) to Healthcheck VNF	Yes	Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation to create OpenStack resources (via openstacksdk/shade)		Yes -NOT an ONAP operations, and is NFVI action, will use dovetail test case, -This could be managed thru MultiVIM test cases if required other than one listed https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation to clean OpenStack resources		Yes -NOT an ONAP operations, and is NFVI action, will use dovetail test case -This could be managed thru MultiVIM test cases if required other than one listed https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Automation to clean ONAP resources (Module, VNF, service instance)		Yes https://wiki.onap.org/display/DW/OVP+LCM+Support	Yes
	Link with VNFREQTS	Yes	Yes	No
Descriptors	Tosca support		Yes	No but reuse of Tosca artifacts provided by SDC for runtime operations
Descriptors	Heat support	Yes	Yes	Yes
CI/CD integration		Yes With jenkins in onapci	NO but Provides API, which could be integrated with CI//CD where ever requried	Yes xtesting-onap with gitlabci for VVP verification on linting + chained-ci for On demand ONAP deployment + test
Results dashboard		Yes grafana dashboard http://onapci.org/grafana	Yes	Yes Based on xtesting, all results pushed to a mongo DB (same than for infrastructure tests) Several web front end (grafana, Kibana, static web pages) used as dahsboards

Testing framework comparison

DRAFT (Table to be completed)