Opened 4 years ago

Closed 3 years ago

#458 closed enhancement (fixed)

Test engine for tickets

Reported by: dmisev Owned by: dmisev
Priority: major Milestone: 9.0
Component: systemtest Version: development
Keywords: Cc: j.yu, pcampalani, abeccati
Complexity: Medium

Description (last modified by dmisev)

Add test engine for testing open bugs reported in tickets, in source:systemtest/testcases_open/test_tickets

The source:systemtest/testcases_open/test_tickets/README summarizes how the test engine works:

====

This test suite includes test cases for bugs reported in tickets. The following 
directories are important for adding tests here:

 - queries  - files containing queries to be tested. Each file should contain 
              one query. The files should be specifically named, as the test 
              execution and reporting depends on this:
              
                NNN-TTT_test-case-name.EXT
              
              where NNN is a test case sequence number, TTT is ticket number.
              EXT indicates the type of the query, and right now can be one of
              
               * rasql
               * wcps (abstract syntax)
               * wcs (KVP URLs)
               * secore (KVP URLs)
               
              Additional bash script can accompany each test case, for any
              extra steps that need to be done before/after running the test
              case, like importing/removing testdata, or for custom check of
              results. The bash scripts should be named:
              
               * NNN-TTT_test-case-name.EXT.pre.sh (run before the test)
               * NNN-TTT_test-case-name.EXT.post.sh (run after the test)
               * NNN-TTT_test-case-name.EXT.check.sh (custom output check)
                
 - oracle   - expected data for testcases, name should be same as the name of
              the corresponding test case, with .oracle appended:
              
                NNN-TTT_test-case-name.EXT.oracle
                
 - testdata - any data used by the tests.
              Testdata from rasql tests (test_grey, test_grey2, test_rgb2)
              and petascope tests (mr, rgb, eobstest) is automatically
              inserted and available in the tests.
 
To run the tests execute test.sh. The test script always returns 0 (successfull
execution), while the summary says that a test is either PASSED if the result
is as expected, or SKIPPED if it failed (ticket still not fixed).

Once a ticket has been fixed and a tests passes, the query in queries should be
renamed to

  NNN-TTT_test-case-name.EXT.fixed

When a query is marked as fixed in this way, the test suite behaves as regular
test suite, i.e. when the test fails the test script exits with an error.

Change History (6)

comment:1 Changed 4 years ago by dmisev

  • Description modified (diff)
  • Status changed from new to accepted

Patch submitted

comment:2 Changed 4 years ago by dmisev

  • Resolution set to fixed
  • Status changed from accepted to closed

comment:3 Changed 4 years ago by abeccati

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:4 Changed 4 years ago by abeccati

  • Status changed from reopened to assigned

Changeset c75cf3eb487edaefdc1946054796e35d61cc52ae removed loading of the mean_summer_airtemp dataset for testing WMS. WMS tests now fail with empty result.
Coverage insertion has to be re-introduced for WMS tests to work.

comment:5 Changed 3 years ago by dmisev

Patch submitted

comment:6 Changed 3 years ago by dmisev

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.