wiki:RasdamanTestSuites

NOTE: this page in an ongoing effort to document the test system used to test rasdaman aka systemtests. Here you will find the overall concept, for further details look for README files within the systemtest directory tree.

Overview

The rasdaman source tree comes with integration tests ("systemtest" for historical reasons") and unit tests.

  • in the directory you have changed code you should check the unit test; in each component directory X there is a subdirectory X/test/ containing the tests.
  • in any case, run the integration test:
    cd systemtest
    make check
    

After your patch submission patchmanager will automatically run systemtest in a sandbox; the result will be flagged in the patchmanager table for each patch submitted. Allow some time (say, 10min) until the result gets visible. Patches which do not pass systemtest will be rejected without further inspection.

Adding a test to the suite

TBD: Howto with a sample test

System test

TODO: this is largely outdated and needs an update.

"make check" in systemtest will automatically find all the tests inside the four test case directories, specifically, testcases_mandatory, testcases_petascope, testcases_fixed and testcases_open.

  1. Whenever a bug is found, there should be a corresponding test be created in the testcases_open directory;
  2. when the bug is fixed, the test should be moved to the testcases_fixed directory;
  3. testcases_petascope holds the test cases for petascope;
  4. testcases_mandatory  holds the mandatory test cases.
  1. Each test should have a folder which is inside one of the above mentioned directories;
  2. the test should be invoked by a shell script inside the folder;
  3. the test result should be told by its EXIT code;
  4. the test can be a pure shell script or a shell script with corresponding test queries and/or test oracles;
  5. test detail should be log in the same folder.

Let us assume that we want to add a test case for our Rasdaman building, you can do as below:

  • how to add a test case?
      1. copy the test package to rasdaman/systemtest/test_open
      2. rename the test packet to test_<testcaseName>
      3. rename the main shell script in rasdaman/systemtest/test_open/test_<testcaseName> to test.sh
    
  • how to add a test query?
      1. save the test query as <queryName>.rasql
      2. copy the query to `rasdaman/systemtest/test_mandatory/test_select/test_rasql
      3. save the related query result as a test oracle with the name <queryName>.rasql.oracle (the same name with test query file)
      4. copy the oracle to rasdaman/systemtest/test_mandatory/test_select/oracle 
    
  • how to generate a test oracle?
      1. check the result correctness 
      2. if the result is a scalar, run
    rasql -q  "$QUERY" --out string|grep Result > oracleName
      3. if the result is an array, run
    rasql -q  "$QUERY" --out file --outfile oracleName
    
  • how to run the test?
      1. run 'make check' under rasdaman/systemtest directory 
    

Queries known to fail

If a query is known to fail and documented by a ticket, it can be marked in the systemtest, so that the result of that query is SKIPPED, rather than FAILED. To do this create a file known_fails (if not yet existing) in the corresponding test dir (next to the test.sh) and put each query file name in a single line in this file.

Petascope tests

IMPORTANT NOTE: Currently the test system is under improvement and the reference test.sh script is the one into test_wcps directory. The development direction is to make that script flexible enough to be directly linked from the other directories (test_wms, test_wcs) and run on the local data on these. This approach is already used in the test_wms part and should be adopted also for the test_wcs as soon as it is proven stable. See also #363

Before submitting any patch which modifies the source of Petascope, WMS, WCS and WCPS smoke/regression tests should be checked before and after applying the changes.

The scripts for WMS, WCS and WCPS testing can be found respectively in:

  • rasdaman/systemtest/testcases_services/test_wcps (reference test.sh implementation)
  • rasdaman/systemtest/testcases_services/test_wcs (test.sh linking to WCPS reference above)
  • rasdaman/systemtest/testcases_services/test_wms (draft, reusing WCPS test.sh)

To run the tests, simply execute in both cases:

$ ./test.sh

from inside the respective directories. Do not execute sh test.sh as the script is written for bash, and sh is often linked to a restricted version of bash like dash, or similar.

Variables like Tomcat port, host, rasdaman connection details, etc. may need to be adapted before running the tests by editing the file:

rasdaman/systemtest/conf/test.cfg

which currently is used for all WMS, WCS and WCPS tests.

It is strongly suggested to additionally run the thread-safety test (by Damiano Barboni) which is still not in the repository, but can by downloaded from ticket #133:

and that you can run with:

$ python TestParallelRequests.py 

Testdata

The following coverages are available for the tests (see rasdaman/systemtest/testcases_services/test_wcps/README):

+--------------+-----+-------+-----------------+------------+-----------------+-----------------+-----------+
| coverage     | dim | type  | pixel extent    |    axes    | geo-boundingbox |  time extension |   CRS     |   
+--------------+-----+-------+-----------------+------------+-----------------+-----------------+-----------+
| rgb          | 2D  | rgb   | 0:399,0:343     | i/j        |        -        |         -       | Index2D   |   
|              |     |       |                 |            |                 |         -       |           |   
| mr           | 2D  | char  | 0:255,0:210     | i/j        |        -        |         -       | Index2D   |   
|              |     |       |                 |            |                 |         -       |           |   
| eobstest     | 3D  | short | 0:5,0:100,0:231 | t/Long/Lat | 25,-40 - 75,75  |  1950-01-01 ->  | Temporal+ |
|              |     |       |                 |            |                 |  -> 1950-01-06  | EPSG:4326 |
| mean_summer_ | 2D  | char  | 0:885,0:710     | Long/Lat   | 111.975,-44.525 |         -       | EPSG:4326 |
| airtemp      |     |       |                 |            | 156.275,-8.975  |         -       |           |   
|              |     |       |                 |            |                 |         -       |           |   
| irr_cube_1   | 3D  | short | 0:99,0:99,0:5   | i/j/k      |        -        |         -       | Index3D   |   
| irr_cube_2   | 3D  | float | 0:62,0:35,0:3   | E/N/ansi   | 75042.72735943, |  2008-01-01 ->  | EPSG:32633|
|              |     |       |                 |            | 5094865.557938- |  -> 2008-01-08  | + ANSI    |   
|              |     |       |                 |            |705042.72735943, |                 |           |   
|              |     |       |                 |            |5454865.5579385  |                 |           |   
+--------------+-----+-------+-----------------+------------+-----------------+-----------------+-----------+

These coverages are automatically inserted if missing.

Adding tests

To add new tests to the test suite, simply add new WCS or WCPS queries to the queries directory. Please adhere to the naming convention, continuing from the last number:

  • WCS KVP : number-meaningful_name.[error.]kvp
  • WCS XML : number-meaningful_name.[error.]xml
  • WCS SOAP : number-meaningful_name.[error.]soap
  • WCS REST : number-meaningful_name.[error.]rest
  • WCPS abstract: number-meaningful_name.[error.]test
  • WCPS XML : number-meaningful_name.[error.]xml
  • rasql : number-meaningful_name.[error.]rasql
IMPORTANT
If the test is meant to raise an exception, add a further .error suffix to the file name before its extension, for both query and oracle.

The associated oracle (<queryname>.oracle) must also be added to the ./oracle directory.
The oracle can be automatically added by running the tests. In this case it can be more convenient to run the tests on the single new query by uncommenting this line in test.sh:

   # uncomment for single test run
   [[ "$f" == 62-* ]] || continue

and choose the proper pattern to select 1 or more tests.

WCS-SOAP test

SOAP request message (

file name: GetCoverage-SOAP.in

)

<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"> 
    <env:Header/>
    <env:Body>
        <wcs:GetCoverage xmlns:wcs="http://www.opengis.net/wcs/2.0"
            xmlns:gml="http://www.opengis.net/gml/3.2"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://www.opengis.net/wcs/2.0 ../../wcsAll.xsd"
            service="WCS" version="2.0.0">
            <wcs:CoverageId>BlueMarbleCov</wcs:CoverageId>
            <wcs:DimensionTrim>
                <wcs:Dimension>Lat</wcs:Dimension>
                <wcs:TrimLow>10</wcs:TrimLow>
                <wcs:TrimHigh>20</wcs:TrimHigh>
            </wcs:DimensionTrim>
            <wcs:DimensionTrim>
                <wcs:Dimension>Long</wcs:Dimension>
                <wcs:TrimLow>10</wcs:TrimLow>
                <wcs:TrimHigh>30</wcs:TrimHigh>
            </wcs:DimensionTrim>
           <gml:format>image/tiff</gml:format>
        </wcs:GetCoverage>
    </env:Body>
</env:Envelope>

Send request by wget:

wget http://ows.rasdaman.org/rasdaman/ows --post-file=GetCoverage-SOAP.in --header="Content-Type: text/xml" -O response.tiff
Last modified 12 months ago Last modified on Oct 9, 2016 8:20:16 AM