Opened 5 years ago

Closed 5 years ago

#172 closed defect (fixed)

WCPS ConstructCoverageExpr

Reported by: pcampalani Owned by: dmisev
Priority: major Milestone: 8.4
Component: petascope Version: 8.3
Keywords: constructcoverageexpr over Cc:
Complexity:

Description

There seems to be a bug when executing a WCPS ConstructCoverageExpr. E.g.:

for cover in (mean_summer_airtemp) 
  return encode(
    coverage temp over $px x (0:0) 
    values 1, 
  "csv")

which throws error in ConstructCoverageExpr.java::buildMetadata() method, at the line:

   xq.getMetadataSource().read(covName);

Stack trace:

SEVERE: InvalidRequest: Coverage 'temp' has invalid metadata
        at petascope.core.DbMetadataSource.read(DbMetadataSource.java:632)
        at petascope.wcps.server.core.ConstructCoverageExpr.buildMetadata(ConstructCoverageExpr.java:158)
        at petascope.wcps.server.core.ConstructCoverageExpr.<init>(ConstructCoverageExpr.java:68)
        at petascope.wcps.server.core.CoverageExpr.<init>(CoverageExpr.java:87)
        at petascope.wcps.server.core.EncodeDataExpr.<init>(EncodeDataExpr.java:60)
        at petascope.wcps.server.core.XmlQuery.startParsing(XmlQuery.java:105)
        at petascope.wcps.server.core.ProcessCoveragesRequest.<init>(ProcessCoveragesRequest.java:98)
        at petascope.wcps.server.core.Wcps.pcPrepare(Wcps.java:167)
        at petascope.wcps.server.core.Wcps.pcPrepare(Wcps.java:161)
        at petascope.wcps.server.servlet.WcpsServlet.doPost(WcpsServlet.java:173)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:679)
Caused by: InvalidRequest: Coverage 'temp' is not served by this server
        at petascope.core.DbMetadataSource.read(DbMetadataSource.java:449)
        ... 35 more

As a temporal workaround, commenting out that line seem to make things work, but this may not be and probably is not an optimal solution.

Change History (11)

comment:1 Changed 5 years ago by pcampalani

Involved WCPS grammar, as currently defined in Petascope:

coverageConstructorExpr returns[CoverageConstructorExpr value]
        : COVERAGE coverage=coverageName OVER ail=axisIteratorList VALUES se=scalarExpr
                { $value = new CoverageConstructorExpr($coverage.value, $ail.value, $se.value); }
        ;   

axisIteratorList returns[AxisIteratorList value]
        : vn=variableName an=axisName LPAREN ie=intervalExpr RPAREN
                { $value = new AxisIteratorList(new AxisIterator($vn.value, $an.value, $ie.value)); }
        (COMMA vn2=variableName an2=axisName LPAREN ie2=intervalExpr RPAREN
                { $value = new AxisIteratorList(new AxisIterator($vn2.value, $an2.value, $ie2.value), $value); })* 
        ;   

scalarExpr returns[ScalarExpr value]
    : e1=metaDataExpr  {  $value = new ScalarExpr($e1.value); }
    | e2=condenseExpr  { $value = new ScalarExpr($e2.value); }
    | e3=booleanScalarExpr   {  $value = new ScalarExpr($e3.value); }
    | e4=numericScalarExpr  {  $value = new ScalarExpr($e4.value); }
    | e5=stringScalarExpr  {  $value = new ScalarExpr($e5.value); }
    | LPAREN e6=scalarExpr RPAREN  { $value = $e6.value; }

intervalExpr returns[IntervalExpr value]
    : lo=indexExpr COLON hi=indexExpr
        { $value = new IntervalExpr($lo.value, $hi.value); }
    | IMAGECRSDOMAIN LPAREN e1=coverageName COMMA e2=axisName RPAREN
        { $value = new IntervalExpr($e1.value, $e2.value); }
    ;   

variableName returns[String value]
        : var=VARIABLE_DOLLAR { $value = new String($var.text); }
        ;   

VARIABLE_DOLLAR: '$'(('a'..'z'|'A'..'Z'|'0'..'9'|'_')*) {setText(getText().substring(1, getText().length())); } ; 

comment:2 Changed 5 years ago by pcampalani

Whereas this query works (after proposed workaround!):

  for c in (mean_summer_airtemp) return encode(
      coverage temp over $px x (0:2) 
      values avg(c[
          x:"CRS:1"(0:1000),
          y:"CRS:1"(0:1000)]), 
  "csv")

  OUTPUT: {48.6546,48.6546,48.6546}

..adding the axis label of the constructed coverage in the subsets, gives error:

1) for c in (mean_summer_airtemp) return encode(
      coverage temp over $px x (0:2)
      values avg(c[
          x:"CRS:1"($px),
          y:"CRS:1"(1000)]), "csv") 

ERROR: 
$ rasql -q "select csv(marray i_i in [0:2] values avg_cells((c) [i_i[0],1000])) from mean_summer_airtemp AS c" --out string
rasql: rasdaman query tool v1.0, rasdaman v8 -- generated on 10.04.2012 15:44:22.
opening database RASBASE at localhost:7001...ok
Executing retrieval query...rasdaman error 353: Execution error 353 in line 1, column 39, near token avg_cells : Operand of quantifier must be multidimensional.
aborting transaction...ok
rasql done. 

2)   for c in (mean_summer_airtemp) return encode(
      coverage temp over $px x (0:2)
      values avg(c[
          x:"CRS:1"($px),
          y:"CRS:1"(0:1000)]), "csv") 

ERROR: line 2:33 no viable alternative at input 'over'
line 5:10 mismatched input 'y' expecting STRING
WCPS: error
WCPS: setting response mimetype to text/html; charset=utf-8
WCPS: returning the following error message
java.lang.NullPointerException
	at petascope.wcps.grammar.EncodedCoverageExpr.<init>(EncodedCoverageExpr.java:40)
	at petascope.wcps.grammar.wcpsParser.encodedCoverageExpr(wcpsParser.java:811)
	at petascope.wcps.grammar.wcpsParser.processingExpr(wcpsParser.java:676)
	at petascope.wcps.grammar.wcpsParser.returnClause(wcpsParser.java:510)
	at petascope.wcps.grammar.wcpsParser.wcpsRequest(wcpsParser.java:219)
	at petascope.wcps.server.core.ProcessCoveragesRequest.abstractQueryToXmlQuery(ProcessCoveragesRequest.java:132)
	at petascope.wcps.server.servlet.WCPSServlet.doPost(WCPSServlet.java:155)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:679)
WCPS: end of error message
WCPS: done with error

comment:3 Changed 5 years ago by olcl

Just to bump this, we are also suffereing with the same bug. Primarily the use of geospatial subsetting in the coverage constructor query.

comment:4 Changed 5 years ago by pcampalani

Hi olcl,
WCPS requests with geographic subsets should work:

   for c in (mean_summer_airtemp) return encode(
       coverage temp over $px x (0:2)
       values avg(c[
           x:"http://www.opengis.net/def/crs/EPSG/0/4326"(115:120),
           y:"http://www.opengis.net/def/crs/EPSG/0/4326"(-40:-35)]),
   "csv")

Response: {0.201255,0.201255,0.201255}.

Assumptions: the native CRS of mean_summer_airtemp is http://www.opengis.net/def/crs/EPSG/0/4326*, and the above-mentioned workaround is applied.

  • Check with:

petascopedb=# select ps_coverage.name, ps_domain.name, ps_crs.name from ps_crsset, ps_crs, ps_domain, ps_coverage where ps_crsset.crs=ps_crs.id and ps_crsset.axis=ps_domain.id and ps_domain.coverage=ps_coverage.id and ps_coverage.id=<coverageName>;

comment:5 Changed 5 years ago by olcl

Hi Piero,

the query i want to use is :

for c in ( myocean_test )
return
  encode( (float)
    coverage histogram over $pt t( 0 : 10 )
    values  avg((c[c[x:"http://www.opengis.net/def/crs/EPSG/0/4326"(-4.552002:-4.552002), y:"http://www.opengis.net/def/crs/EPSG/0/4326"(53.904338:53.904338),t($pt)] > 0.0) * (c[c[x:"http://www.opengis.net/def/crs/EPSG/0/4326"(-4.552002:-4.552002), y:"http://www.opengis.net/def/crs/EPSG/0/4326"(53.904338:53.904338),t($pt)])),
    "csv")

where i uses geographic susetting within a coverage constructor. Does the proposed work around fix this?

The current error i get is :

line 5:23 no viable alternative at input 'over'
line 6:90 mismatched input 'y' expecting STRING
WCPS: error
WCPS: setting response mimetype to text/html; charset=utf-8
WCPS: returning the following error message
java.lang.NullPointerException
	at petascope.wcps.grammar.EncodedCoverageExpr.<init>(EncodedCoverageExpr.java:40)
	at petascope.wcps.grammar.wcpsParser.encodedCoverageExpr(wcpsParser.java:811)
	at petascope.wcps.grammar.wcpsParser.processingExpr(wcpsParser.java:676)
	at petascope.wcps.grammar.wcpsParser.returnClause(wcpsParser.java:510)
	at petascope.wcps.grammar.wcpsParser.wcpsRequest(wcpsParser.java:219)
	at petascope.wcps.server.core.ProcessCoveragesRequest.abstractQueryToXmlQuery(ProcessCoveragesRequest.java:132)
	at petascope.wcps.server.servlet.WCPSServlet.doPost(WCPSServlet.java:155)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:679)
WCPS: end of error message
WCPS: done with error

comment:6 Changed 5 years ago by anonymous

the above has a typo in, the query should have read :

for c in ( myocean_test )
return
  encode( (float)
    coverage histogram over $pt t( 0 : 10 )
    values  avg((c[x:"http://www.opengis.net/def/crs/EPSG/0/4326"(-4.552002:-4.552002), y:"http://www.opengis.net/def/crs/EPSG/0/4326"(53.904338:53.904338)] > 0.0) * (c[c[x:"http://www.opengis.net/def/crs/EPSG/0/4326"(-4.552002:-4.552002), y:"http://www.opengis.net/def/crs/EPSG/0/4326"(53.904338:53.904338)])),
    "csv")

sorry

comment:7 Changed 5 years ago by pcampalani

Hi,
firstly, the hereby proposed workaround seems to create leaks in other WCPS requests which do not involve a ConstructCoverage expression (Jelmer's WCPS client started having problems). I couldn't verify the actual effects of that workaround for general WCPS queries.

Anyway, your syntax seems fine, a part maybe from an other typo you did (... 0.0) * (c[c[x: ...).
You are also requesting slices, not trims, so you may remove that duplicated second extent in each subset.

I will make some test locally, hope to get back with some news.
If you want to try out the workaround, just contact me, I will provide a .war for Petascope to try out.

comment:8 Changed 5 years ago by pcampalani

  • Priority changed from minor to major

comment:9 Changed 5 years ago by pcampalani

First of all we should mind the ticket #64, so that in the ConstructCoverage we should force subsetting of time as well: t($pt:$pt) instead of simply t($pt), not to mix slices with trims inside the same square brackets.

You can appreciate results on a single timestamp with the following query (I simplified it a little for readability):

for c in ( myocean_test ) return encode( (float) 
        avg(c[
            x:"http://www.opengis.net/def/crs/EPSG/0/4326"(-4.552002:-4.552002),
            y:"http://www.opengis.net/def/crs/EPSG/0/4326"(53.904338:53.904338),
            t:"CRS:1"(5:5)]),
    "csv")

Instead, when wanting to construct an histogram of values, there is a bug in the parser, before looking at any metadata:

for c in ( myocean_test ) return encode( (float)
    coverage histogram over $pt t( 0 : 10 ) values    
        avg(c[
            x:"http://www.opengis.net/def/crs/EPSG/0/4326"(-4.552002:-4.552002),
            y:"http://www.opengis.net/def/crs/EPSG/0/4326"(53.904338:53.904338),
            t:"CRS:1"($pt:$pt)]),
    "csv")

gives (similarly to ticket #64) :

SEVERE: line 2:24 no viable alternative at input 'over'
SEVERE: line 5:10 mismatched input 'y' expecting STRING
INFO: WCPS: error

comment:10 Changed 5 years ago by pcampalani

I might have found a workaround.
Firstly:

Lemma: references to a variable $var defined in the ConstructCoverageExpr can be put only inside of slices operations (no trims).

Then, since mixing of trims and slices within a bracket subsetting is currently not allowed (ticket #64), an average over different time slides and over a restricted spatial domain could be achieved this way:

for c in (MODIS_33N_2010170_WGS84) return encode((float)
      coverage temp over $t t (0:3)
      values avg(slice(
        trim(c, {
          x(10:11),
          y(46:47)}), {
        t($t)})),
  "csv")

This works when commenting out the reported line in ConstructCoverageExpr.java::buildMetadata(), and this however seems to create problems with other WCPS expressions; going to search for a better solution.

comment:11 Changed 5 years ago by dmisev

  • Resolution set to fixed
  • Status changed from new to closed

I believe we fixed this in the latest sources

Note: See TracTickets for help on using tickets.