Opened 21 months ago

Closed 6 months ago

#941 closed defect (fixed)

Use wcst_import in OGC systemtest

Reported by: dmisev Owned by: bphamhuu
Priority: major Milestone: 9.2
Component: systemtest Version: development
Keywords: Cc: mdumitru, vmerticariu
Complexity: Medium

Description

We should switch to wcst_import for importing the testdata.

Attachments (1)

rgb_import.json (589 bytes) - added by bphamhuu 18 months ago.

Download all attachments as: .zip

Change History (27)

comment:1 Changed 21 months ago by dmisev

  • Milestone changed from 9.1 to 9.2

comment:2 Changed 18 months ago by dmisev

  • Owner changed from dmisev to bphamhuu
  • Status changed from new to assigned

comment:3 Changed 18 months ago by dmisev

So you should open source:systemtest/util/petascope.sh and there you have the various import_* scripts that ingest the data used in the petascope (WCS/WCPS/..) tests.

In these functions we should replace the calls to rasql and psql with wcst_import.sh and an ingredient file for each coverage (rgb, mr, mean_summer_airtemp, etc).

comment:4 Changed 18 months ago by bphamhuu

Ok, Dimitar. Now, I know how to do this and could manually inject my script to petascope.sh. However, it looks like something error with WCST when I try to import rgb with recipe (map_mosaic). Why it need to -0.5 in i and j axis?

Running
Progress: [------------------------------] 0/1 0.00% 
Exception thrown when trying to insert slice: 
{Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}
Retrying, you can safely ignore the warning for now. Tried 1 times.

Progress: [------------------------------] 0/1 0.00% 
Failed to insert slice. Attempted 1 times.
Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=UpdateCoverage&coverageId=rgb1&subset1=i(0.5,399.5)&inputCoverageRef=file:///tmp/9154dae2_b18a_4c12_8bea_5516a0b71d11.gml&subset2=j(344.5,-0.5)
Error Code: RuntimeError
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>Runtime error while processing request</ows:ExceptionText>
    </ows:Exception>
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>petascope.wcps2.error.managed.processing.UnorderedSubsetException : Invalid subsetting coordinates: 344.5:-0.5 for axis j.</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

comment:5 Changed 18 months ago by dmisev

Do you use crs Index2D?

Changed 18 months ago by bphamhuu

comment:6 Changed 18 months ago by bphamhuu

Have a look in here Dimitar: http://rasdaman.org/attachment/ticket/941/rgb_import.json, I think there are some other problem because with this recipe I could ingest another files .Tiff.

comment:7 follow-up: Changed 18 months ago by dmisev

  • Cc vmerticariu added

Hm not sure if subset_correction should be false?

comment:8 in reply to: ↑ 7 Changed 18 months ago by mdumitru

Yes, it should be false, that should be enabled only for extra small offset vectors.

comment:9 Changed 18 months ago by bphamhuu

Thanks guys. I've passed over this error and I could ingest RGB to Rasdaman by Petascope. Once I also have a trouble with this ingestion due to the permit of RASBASE (I've changed it again for rasdaman user instead of only root can write). So I think the exception could be add some information like ("Check the permission of RASBASE" as suggestion will help users).

Initialization
Collected files: ['/home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png']...

Validation
The recipe has been validated and is ready to run.
Recipe: map_mosaic
Coverage: png
WCS Service: http://localhost:8080/rasdaman/ows
Operation: INSERT
Subset Correction: False
Mocked: False
Track files: True
WMS Import: False

A couple of files have been analyzed and valid to import in below. Please check the axis subsets are correct.

Slice 1: {Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}

Press Enter to Continue...: 

Running
Progress: [------------------------------] 0/1 0.00% Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=InsertCoverage&coverageRef=file:///tmp/b03ad296_56e5_4728_963b_f5d7d8f76b25.gml&tiling=ALIGNED [0:100, 0:100]&pixelDataType=Byte&useId=existing
Error Code: RasdamanRequestFailed
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RasdamanRequestFailed">
        <ows:ExceptionText>Error evaluating rasdaman query: 'CREATE COLLECTION png RGBSet</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

comment:10 Changed 18 months ago by bphamhuu

Hi again. I have messed up with the permission so I tried to reinstall Rasdaman and retry with the sample data. But this time with "subset_correction": false, it has an error same to the old error.

Slice 1: {Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}

Press Enter to Continue...: 

Running
Progress: [------------------------------] 0/1 0.00% 
Exception thrown when trying to insert slice: 
{Axis Subset: i(0.0,400.0) j(344.0,0.0) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png}
Retrying, you can safely ignore the warning for now. Tried 1 times.

Progress: [------------------------------] 0/1 0.00% 
Failed to insert slice. Attempted 1 times.
Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=UpdateCoverage&coverageId=rgb&subset1=i(0.0,400.0)&inputCoverageRef=file:///tmp/f242a470_d35c_4be8_b451_46b1502cf4ec.gml&subset2=j(344.0,0.0)
Error Code: RuntimeError
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>Runtime error while processing request</ows:ExceptionText>
    </ows:Exception>
    <ows:Exception exceptionCode="RuntimeError">
        <ows:ExceptionText>petascope.wcps2.error.managed.processing.UnorderedSubsetException : Invalid subsetting coordinates: 344.0:0.0 for axis j.</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

Configuration file:

{

  "config": {

    "service_url": "http://localhost:8080/rasdaman/ows",

    "tmp_directory": "/tmp/",

    "crs_resolver": "http://localhost:8080/def",

    "default_crs": "http://localhost:8080/def/crs/OGC/0/Index2D",

    "mock": false,

    "automated": false,

    "subset_correction": false

  },

  "input": {

    "coverage_id": "rgb", 

    "paths": [

      "/home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/rgb.png"

    ]

  },

  "recipe": {

    "name": "map_mosaic", 
    "options": {
 	  "tiling": "ALIGNED [0:100, 0:100]"
    }

  }

comment:11 Changed 18 months ago by dmisev

Yes 344.0:0.0 doesn't look correct, the j is 344 pixels so I'd expect the subset to be 343:0

comment:12 Changed 18 months ago by bphamhuu

ok, so Dimitar, you know I'm stucking at this problem but I think when I still could ingest a tif file so let consider it an example of 'rgb' and I will finish the rest now (I assume it ingest data good) and now what I consider is check that coverage ID is existed on Petascope and if it existed then just ignore importing.

I think I should keep the way to check Coverage ID is existed by Postgresql (because when I try with "wget" to Petascope with DescribeCoverage? service (even it is existed or not) it only return 500 and stuck.

What about your idea? Should we just import by Petascope and keep check exist and delete by PSQL?

comment:13 Changed 18 months ago by dmisev

It would be good to get away from using low-level postgresql checks as much as possible.

If DescribeCoverage fails for you then something is wrong in your petascope.

comment:14 Changed 18 months ago by bphamhuu

Ok, I got the idea and you also agree like this. I want to reconfirm that petascope has no error when I could DescribeCoverage? on Web browser http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=rgb it return the GML with full detail and no problem (even when the coverage ID is not existed), like this

<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" version="2.0.0" xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd">
<ows:Exception locator="rgb2" exceptionCode="NoSuchCoverage">
<ows:ExceptionText>
One of the identifiers passed does not match with any of the coverages offered by this server
</ows:ExceptionText>
</ows:Exception>
</ows:ExceptionReport>

However, we need to check as you said that the Coverage is existed without PSQL so the only way is access HTTP Request by Terminal and the most common too lis "wget" return about the error of Server. Please view below and yes the reason is of course from Server but it is quite hard to find out the reason when WebBrowser? works good and Terminal stop.

wget http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=rgb2
[1] 18364
[2] 18365
[3] 18366
[2]-  Done                    version=2.0.1
[3]+  Done                    request=DescribeCoverage
[rasdaman@gonzo test_wcps]$ --2015-10-14 16:55:07--  http://localhost:8080/rasdaman/ows?service=WCS
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8080... connected.
HTTP request sent, awaiting response... 500 Server Error
2015-10-14 16:55:07 ERROR 500: Server Error.

comment:15 Changed 18 months ago by dmisev

Can you try

wget "http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=rgb2"

comment:16 Changed 18 months ago by dmisev

In general, always quote arguments in bash.

comment:17 Changed 18 months ago by bphamhuu

ok, it is the key, it works with error 404 (NOT found) if coverage is not existed and download the GML file if it is exist. Thanks.

comment:18 Changed 18 months ago by bphamhuu

Ok, it looks good now and import also success with check exist before importing. Do you see anywhere we need deleveCoverage? I don't see because if coverage is exist then it just ignore or if not exist then it will insert.

I ask this because I see some old idea before:

  if [ $? -ne 0 ]; then
      #drop_colls $COLL
      #drop_petascope $COLL
      log "  Coverage ID: $COLL is not exist, now importing $COLL... 

and not really sure why if coverage is not exist, still have to delete it Petascope.

comment:19 Changed 18 months ago by bphamhuu

I could not know why not only PNG file but I could not import file eobs.nc and mean_summer_airtemp.tif. It is the same errors with the source domain is not fit with target domain (I've change coverage id to new id everytime and also subset_correction: true).

I could only ingest irr_cube_2 files by time_serries_irregular recipe (and with my test rgb and mr_1 (tif not png file because I have problem as previous comments)).

Can I get from Alex also, what should I do to import PNG (rgb, mr) and mean_summer_airtemp and eobs with WCST-Import?

Here is error with mean_summer_airtemp and eobs

wcst_import.sh '/home/rasdaman/ingredient.txt'
Initialization
Collected files: ['/home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/mean_summer_airtemp.tif']...

Validation
The recipe has been validated and is ready to run.
Recipe: map_mosaic
Coverage: ads8
WCS Service: http://localhost:8080/rasdaman/ows
Operation: INSERT
Subset Correction: True
Mocked: False
Track files: True
WMS Import: False

A couple of files have been analyzed and valid to import in below. Please check the axis subsets are correct.

Slice 1: {Axis Subset: i(111.975,156.275) j(-44.525,-8.975) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/mean_summer_airtemp.tif}

Press Enter to Continue...: 

Running
Progress: [------------------------------] 0/1 0.00% 
Exception thrown when trying to insert slice: 
{Axis Subset: i(111.975,156.275) j(-44.525,-8.975) 
Data Provider: file:///home/rasdaman/rasdaman/rasdaman/systemtest/testcases_services/test_wcps/testdata/mean_summer_airtemp.tif}
Retrying, you can safely ignore the warning for now. Tried 1 times.

Progress: [------------------------------] 0/1 0.00% 
Failed to insert slice. Attempted 1 times.
Service Call: http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=UpdateCoverage&coverageId=ads8&subset1=i(112.0,156.25)&inputCoverageRef=file:///tmp/f7066bf8_18fa_482d_be05_c257af718a01.gml&subset2=j(-44.55,-8.95)
Error Code: RasdamanError
Error Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ows:ExceptionReport version="2.0.0"
    xsd:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd"
    xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
    <ows:Exception exceptionCode="RasdamanError">
        <ows:ExceptionText>rasdaman error 967: Update error 967 in line 1, column 1, near token UPDATE: Source MDD domain is not within target update MDD domain.
</ows:ExceptionText>
    </ows:Exception>

</ows:ExceptionReport>

comment:20 Changed 18 months ago by bphamhuu

ok, as ticket 1018 has fixed, now I could ingest all the images without georeferenced with axis (integer point). But still stuck at error 967 when try images with float points in each corners.

After few hours of debugging I can conclude there are something wrong with "ungeoreferenced images" that origins are float value).

But I could not wait until have an answer (for example wait until this is fixed correctly), so I just using gdal tool to set georeference to 2 test files: eobs.nc (netCDF) and mean_summer_airtemp (TIF) and now I could impost by WCST-Tool to Petascope.

So now, I assume everything is working and will try to finish this task soon.

comment:21 Changed 17 months ago by bphamhuu

This ticket could not finished soon because:

+ WCST_import could not decode 1D (I think should add function to import CSV file for float_1d).
+ There are differences cases between WCST_Import and test_oracle (query by rasql) with non-georefenced like rgb, mr as ticket 1021 has shown.
+ eobstest also has big different with test oracles and WCST_Import (in test oracles it is a 3D cube (Lat, Long, Time - each band is 1 time anchor) but import by WCST_Import it is 1 2D with 5 bands).
+ Need effor to create float_4d and irr_cub_1.

So this can only change in long term, even need to modify a lot of test oracles (a new test suite can be created) as Vlad suggested like eobstest is deprecated.

comment:22 Changed 17 months ago by mdumitru

Good overview Bang. I propose the following things:

@Dimitar: What do you think?

comment:23 Changed 17 months ago by dmisev

Agreed; it would be good to have a floating point dataset as well.

eobstest is a couple of slices of a real dataset btw and is in netcdf format, I think it's not bad to have it as well.

comment:24 Changed 17 months ago by mdumitru

@Dimitar: Didn't know that about eobs. I see on gdal that 6 bands are listed and 2 dimensions, while ncdump lists 3 dimensions and 4 variables. It seems we should provide a NetcdfMetadataProvider? besides a GdalMetadataProvider? in WCSTImport as NetCDF support seems abysimal in GDAL. Maybe we can setup a meeting and discuss what we can do for NetCDF support, I'm not that familiar with the format?

comment:25 Changed 17 months ago by dmisev

Yes GDAL is doing some workarounds to fit the netcdf model in their framework I guess, which doesn't really work out that well. There is python API for netcdf, which could be used in wcst_import?

comment:26 Changed 6 months ago by dmisev

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.