Opened 4 years ago

Closed 4 years ago

#363 closed task (fixed)

Enhancement of petascope regression test suite

Reported by: abeccati Owned by: nkolev
Priority: major Milestone: 8.5
Component: systemtest Version: 8.4
Keywords: Cc: dmisev, pcampalani, arezaeim, j.yu
Complexity: Medium

Description (last modified by abeccati)

As part of the proposed projects we have regression test refactoring

Wrt regression testing we need improvement especially with petascope tests (documented here)

Most compelling is the inclusion of an oracle or similar structure for evaluating query output and enable also queries that purposefully generate errors to be part of successful test cases.

Currently the test system is under improvement and the reference test.sh script is the one into test_wcps directory. The development direction is to make that script flexible enough to be directly linked from the other directories (test_wms, test_wcs) and run on the local data on these. This approach is already used in the test_wms part and should be adopted also for the test_wcs as soon as it is proven stable.

Change History (18)

comment:1 Changed 4 years ago by dmisev

Note that I have submitted some systemtest refactoring patch, you should do your work after applying that on.

comment:2 Changed 4 years ago by dmisev

Note I've updated a bit the test page at wiki:RasdamanTestSuites#Petascopetests

The configuration file for petascope and the whole systemtest now is source:systemtest/conf/test.cfg

Last edited 4 years ago by pcampalani (previous) (diff)

comment:3 Changed 4 years ago by pcampalani

As soon as the oracle-based testing system is set up, as a next step we should add (many) error-prone queries and verify they actually return an error/exception (this will help us reviewing error messages and identifying cumbersome ones, e.g. see #394).

comment:4 Changed 4 years ago by dmisev

That should go in another ticket Piero, I don't think Nikolce has the background to add such tests.

comment:5 Changed 4 years ago by pcampalani

Yes, it was general next-step, not Nikolce's next step :)
It was a note, then we can either create a new ticket or go on with this one, whatever.

comment:6 Changed 4 years ago by abeccati

  • Status changed from new to assigned

I strongly second Piero's proposal and created #396 for that purpose (we should keep tasks/ticket as "atomic" tasks as much as we can to ease management)
Let's finalize infrastructure first with this ticket.

@nkolev: Please "accept" the ticket (accept option at bottom of modify ticket) so we can track active assignment.

comment:7 Changed 4 years ago by nkolev

  • Status changed from assigned to accepted

comment:8 Changed 4 years ago by dmisev

patch submitted

comment:9 Changed 4 years ago by pcampalani

Missing ref from comment:8 : changeset:a6625cd.

Now WCS tests are left.

comment:10 Changed 4 years ago by abeccati

  • Complexity changed from Easy to Medium
  • Description modified (diff)

Also WMS tests are missing, basic ones should be included. Meanwhile patch submitted to master (2013-07-23 14:02:17) for WMS inclusion: links directly to the test.sh engine script.

comment:11 Changed 4 years ago by abeccati

Included first test passing getCapabilities and loading WMS test data "mean_summer_airtemp" as laid out by Piero at http://rasdaman.org/wiki/WmsImportTools
Please note that the WMS engine is the same as the wcps one (to be used as refernce) and the patch (2013-07-23 19:31:01) extends it to detect service name according to the base directory where it is linked from (e.g. test_wms). This can be reused for all services to ease maintenance.

comment:12 Changed 4 years ago by abeccati

  • Cc j.yu added
  • Description modified (diff)
  • Milestone set to 8.5
  • Status changed from accepted to assigned

Changing to assigned since some changes have been committed meantime and no active progress seen in some time. nkolev please re-accept when working on it again and read the updated doc at RasdamanTestSuites to get aligned.
Adding also Jin (systemtests component owner) in cc for overall test systems coordination.

comment:13 Changed 4 years ago by nkolev

  • Status changed from assigned to accepted

comment:14 Changed 4 years ago by dmisev

Alan it seems like your patch has broken the WCPS test, please check. E.g.

src/systemtest/testcases_petascope/test_wcps$ ./test.sh 
SCRIPT_DIR is set to: systemtest/testcases_petascope/test_wcps
Old logfile found, copying it to systemtest/testcases_petascope/test_wcps/log.save
test.sh: starting test at Sat Aug  3 19:20:43 CEST 2013
test.sh: 
Service name is: wms
Service endpoint is set to: http://localhost:8080/petascope/wms
test.sh: rgb already imported.
test.sh: mr already imported.
test.sh: eobstest already imported.
test.sh: mean_summer_airtemp already imported.

--------------------------------------------------------
running test: 01-just_trim.test

for c in ( rgb ) return encode( c[ x(0:100), y(0:100) ], "png" )

KVP request: http://localhost:8080/petascope/wms?for c in ( rgb ) return encode( c[ x(0:100), y(0:100) ], "png" )


real	0m0.698s
user	0m0.004s
sys	0m0.008s
test.sh: byte comparison
/home/dimitar/projects/rasdaman/community/src/systemtest/testcases_petascope/test_wcps/oracle/01-just_trim.oracle.data /home/dimitar/projects/rasdaman/community/src/systemtest/testcases_petascope/test_wcps/output/01-just_trim.test.out differ: byte 1, line 1
test.sh:  ->  QUERY FAILED

Also not clear why is SCRIPT_DIR printed.

comment:15 Changed 4 years ago by dmisev

test.sh:60 has

SVC_NAME=$(echo test_wms | cut -d "_" -f 2)

comment:16 Changed 4 years ago by dmisev

Ok I submitted patch that fixes it.

comment:17 Changed 4 years ago by dmisev

  • Priority changed from critical to major

comment:18 Changed 4 years ago by dmisev

  • Resolution set to fixed
  • Status changed from accepted to closed

Actually closing this; for further issues please open new ticket.

Note: See TracTickets for help on using tickets.