Page Comparison

...

• User Management
  • Users log in through a Linux Foundation Open ID
  • A user logging in for the first time is automatically assigned the "user" role.
  • User Roles
    •  "user"
      • Can upload and manage test results
      • Can only see own test results
      • Can create an application to submit their results to review
    • "reviewer"
      • can see all test results marked as "for review" by its "user" (the user is the owner of the results / application)
    • "admin"
      • can manage assigned user roles
      • can manage (create, update, delete) entries to the marketplace
  • A portal user can have multiple assigned roles, i.e. Jo can be assigned the roles of "admin" and "user"
• Results Format
  • Should be as flexible as possible.
  • Results are uploaded as a zip or tar.gz file, other formats will be rejected
  • Must include a "test result summary" in the archive file root
  • The "test results summary" will include:
    • Version of the tool used (i.e. what version of functest was running)
    • Date & time of the test run
  • validation and display of test results (see also terminology below)
    • the web portal must validate uploaded test results by comparing the "test result summary" to a "test result guideline"
      • "test result guideline": source of truth
        • list of all test cases which are part of a given OVP release
          • use case: detect if test cases are missing from uploaded test results
        • the expected result for passing each test case (functional tests: "pass", non-functional: "value")
        • stored in web portal only
      • "test result summary"
        • part of the "test result package" generated by test tool
        • json formatted
        • should include in addition to today (AP on test tooling team)
          • OVP release ID (e.g. 2020.10)
          • OVP program type (e.g. NVFI, VNF, ...)
        • example of a "test result package" currently generated by test tooling:
          • http://artifacts.opnfv.org/functest/DUXGBAW88H7Q.zip
    • optional requirements, requires close collaboration with and input from test tooling team
      • define a schema for formal validation of test result summary
      • define a schema for formal validation of test result guide
• Validation of Results
  • The portal should be capable of validating the submitted results.
  • Validation checks the results contain the correct test cases (minimum set) and those test cases pass
  • The set of test cases (minimum set) should be controlled by the portal admin.
    • An OVP release may include multiple "minimum sets" that apply to different releases of Functest and OpenStack

• Terminology
  • "test result package": archive containing "test result summary" file and individual logs
  • "test result summary": json formatted file containing a summary of all test cases / one run of the compliance test tool
  • "test result guideline": json formatted file containing all tests which are part of an OVP release + expected result for passing a test

...