Opened 8 years ago

Closed 3 years ago

#15 closed enhancement (fixed)

condense max and min bug

Reported by: dirk.daems@… Owned by: j.yu
Priority: critical Milestone: 8.5
Component: qlparser Version: 8.0
Keywords: condense construct Cc: dmisev, vmerticariu, mdumitru
Complexity: Hard

Description (last modified by dirk.daems@…)

Doesn't seem to work in the pre-release of v9.0.
Got the following exception using the last WCPS query listed in this ticket:

WcpsError: Invalid coverage Expression, next node: condense - Invalid coverage Expression, next node: condense
        at petascope.wcps.server.core.CoverageExpr.<init>(CoverageExpr.java:171)
        at petascope.wcps.server.core.ConstructCoverageExpr.<init>(ConstructCoverageExpr.java:87)
        at petascope.wcps.server.core.CoverageExpr.<init>(CoverageExpr.java:94)
        at petascope.wcps.server.core.EncodeDataExpr.<init>(EncodeDataExpr.java:77)
        at petascope.wcps.server.core.XmlQuery.startParsing(XmlQuery.java:120)
        at petascope.wcps.server.core.ProcessCoveragesRequest.<init>(ProcessCoveragesRequest.java:102)
        at petascope.wcps.server.core.Wcps.pcPrepare(Wcps.java:118)
        at petascope.wcps.server.core.Wcps.pcPrepare(Wcps.java:112)
        at petascope.wcps.server.servlet.WcpsServlet.doPost(WcpsServlet.java:181)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:722)

System tests involving condense min / max also fail.

Change History (25)

comment:1 Changed 7 years ago by cj

  • Description modified (diff)
  • Resolution set to fixed
  • Status changed from new to closed

there is no such operator max or min (i.e. writing max(1,2) will also not be recognized). So the problem is not in the condenser per se but in the grammar. Closing the ticket.

comment:2 Changed 7 years ago by pbaumann

  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Type changed from defect to enhancement

the documentation in the QL Guide indeed lists these operations, so they should be supported.

comment:3 Changed 6 years ago by dmisev

  • Owner set to dmisev
  • Status changed from reopened to accepted

comment:4 Changed 5 years ago by pbaumann

  • Milestone set to 9.0
  • Owner changed from dmisev to j.yu
  • Status changed from accepted to assigned

comment:5 follow-up: Changed 4 years ago by pcampalani

  • Complexity set to Very Hard

Hi there,

in the meantime, a workaround for WCPS condense expressions with min and max -- leading to this unsupported RasQL condense operations -- is achieved by either using a direct min() operator call, or either, in case of a condense operation nested into a coverage constructor, replacing the nested condensation with a min(<coverageConstr>).
E.g.

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   min(
        coverage mincov
        over     $py y (300:301)
        values   c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

Instead of:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        values   c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

comment:6 Changed 4 years ago by pcampalani

  • Complexity changed from Very Hard to Medium

comment:7 Changed 4 years ago by abeccati

  • Milestone changed from 9.0 to 8.5

comment:8 Changed 4 years ago by j.yu

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:9 Changed 4 years ago by dmisev

  • Resolution fixed deleted
  • Status changed from closed to reopened

Seems like it was fixed in rasql but not yet in WCPS.

comment:10 in reply to: ↑ 5 Changed 4 years ago by dmisev

  • Resolution set to fixed
  • Status changed from reopened to closed

Replying to pcampalani:

Instead of:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        values   c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

Actually it's working in WCPS too, note that condense should have using, not values. I've added a test in the WCPS tests.

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        using    c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

comment:11 Changed 4 years ago by dirk.daems@…

  • Description modified (diff)
  • Reporter changed from j.yu@… to dirk.daems@…
  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:12 follow-up: Changed 4 years ago by pcampalani

Hi Dirk,

here I get the query working, e.g.:

for c in (mr)
  return encode(
     coverage mincov
     over     $px x (50:61)
     values   (
        condense max
        over     $py y (0:100)
        using    c[i:"CRS:1"($px), j:"CRS:1"($py)]
     )
  ), "csv")

{84,83,89,87,85,85,85,87,80,80,74,80}

Which query did you run?
Did you use CRS axis labels? (mind i and j in the query)

Last edited 4 years ago by pcampalani (previous) (diff)

comment:13 in reply to: ↑ 12 Changed 4 years ago by dirk.daems@…

Replying to pcampalani:

Hi Dirk,

here I get the query working, e.g.:

for c in (mr)
  return encode(
     coverage mincov
     over     $px x (50:61)
     values   (
        condense max
        over     $py y (0:100)
        using    c[i:"CRS:1"($px), j:"CRS:1"($py)]
     )
  ), "csv")

{84,83,89,87,85,85,85,87,80,80,74,80}

Which query did you run?
Did you use CRS axis labels? (mind i and j in the query)

Piero,

I used the query in this ticket:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px x (100:101)
     values   (
        condense min
        over     $py y (300:301)
        using    c[x:"CRS:1"($px), y:"CRS:1"($py)]
     )
  ), "csv")

comment:14 follow-up: Changed 4 years ago by pcampalani

Hi Dirk,
that's the point: that query is not valid in v9.0 since the axis labels are wrong. OGC:Index2D CRS defines two axes named i and j, not x and y.
Please checkout the user guide for further explanations, and close the ticket if you can run the same query I put in comment:13, thanks.

comment:15 in reply to: ↑ 14 Changed 4 years ago by dirk.daems@…

Replying to pcampalani:

Hi Dirk,
that's the point: that query is not valid in v9.0 since the axis labels are wrong. OGC:Index2D CRS defines two axes named i and j, not x and y.
Please checkout the user guide for further explanations, and close the ticket if you can run the same query I put in comment:13, thanks.

Piero,

Even when replacing 'x' and 'y' by 'i' and 'j' I get the same error. I also tried your query in comment 13, but with the exact same result. Yesterday I downloaded and compiled the latest master (9.0.0-rc1 experienced some issues according to Dimitar) so I have a recent Rasdaman version. FYI: systemtest 84 which also uses a 'condense min' statement also fails.
What I find strange is that your query still uses "CRS:1" while this is not present in the ps_crs table -> shouldn't that be e.g. "http://<servername>:8080/def/crs/OGC/0/Index2D" then?

Regards,
Dirk

comment:16 Changed 4 years ago by pcampalani

Hi Dirk,
I cannot reproduce your problem unluckily, since here both query in comment:13 and WCPS systemtest n.84 work fine.

I suggest we move to the m-list to further inspect the issue (refer to the ticket number in the subject), I'd need your server logs and the HEAD of your repo.

Regarding CRS:1: this is (still) our way to access internal grid coordinates. Index CRSs are external indexed coordinate systems which we assign to non-georeferenced coverages, but there can be a difference there.

If you look in out systemtest import functions, you see we attach Index CRSs to mr and rgb. The orientiation of such CRSs is not the same as the internal (rasdaman) CRS:1 since the latter starts from UL corner of the image, whereas the CRS sets its origin (0,0) to the LL corner, which is more natural: increasing vertical coordinates go upwards.

In practice this means that if you ask subset=j(0) in a WC*S request you get the bottom line (CRS:1 index = hi).

One can always sync the external Index CRS to have the same orientation as the internal rasdaman space: this is done for irr_cube_1, where the CRS origin is placed exactly where the rasdaman origin is, so UL. irr_cube_1 is a fake image, with no meaningful values, but with a photo for example, doing so would mean that you see the photo flipped upside down.

comment:17 Changed 4 years ago by dirk.daems@…

  • Resolution set to fixed
  • Status changed from reopened to closed

Piero,

I forgot to change the axisnames x and y to Long and Lat resp.
The following query seems to work:

for c in (mean_summer_airtemp)
  return encode(
     coverage mincov
     over     $px i (100:101)
     values   (
        condense min
        over     $py j (300:301)
        using    c[Long:"CRS:1"($px), Lat:"CRS:1"($py)]
     )
  ), "csv")

Please note that the errormessage ("WCPS Error: Invalid coverage Expression, next node: condense - Invalid coverage Expression, next node: condense") didn't guide me to change the axis names. Seems like a valuable improvement to me.

Regards,
Dirk

comment:18 Changed 4 years ago by pbaumann

Dirk, you are right - insufficient error message details are a known issue. Actually, an alpha version of WCPS 2 is already in the repo, with a substantially improved error handling. So this will change soon.

comment:19 Changed 4 years ago by pcampalani

Thanks for the feedback Dirk, indeed we are waiting for WCPS 2.0 for a thorough improvement of error handling. There has been effort on this in the past (#206), but in the end the task could not be achieved.

Meanhwile you might have a look at this discussion I started in the m-list: you can find some general suggestion on how to deal with WCPS query construction.

comment:20 Changed 3 years ago by pcampalani

  • Cc dmisev vmerticariu mdumitru added
  • Complexity changed from Medium to Hard
  • Keywords condense construct added
  • Priority changed from major to critical
  • Resolution fixed deleted
  • Status changed from closed to reopened

While the WCPS runs without error, the retrieved values are wrong: it seems that the parametrized indexes gets lost when a condense RasQL expression is placed inside a coverage contructor.

The query proposed by Dirk in comment:17 correctly translates to:

select csv(
  marray i_i in [100:101] values
  condense min over i_j in [300:301] using (c) [i_i[0],i_j[0]]
) from mean_summer_airtemp AS c 

But instead of {90,91} (see equivalent simplified query below), the extracted cells are {0,0}, which probably means that '(c) [i_i[0],i_j[0]]' is internally translated to '(c) [0:0,0:0]'.

Simplified query (workaround when condense is not strictly necessary):

for c in (mean_summer_airtemp)
return encode(
   coverage mincov
   over     $px i (100:101)
   values   min(c[Long:"CRS:1"($px), Lat:"CRS:1"(300:301)]
)), "csv") 

Note: this ticket holds for current development too (pre-9.0 release).

comment:21 Changed 3 years ago by pcampalani

More flexible workaround is in comment:5, which for release 9.0 translates to:

for c in (mean_summer_airtemp)
return encode(
   coverage mincov
   over     $px x (100:101)
   values   min(
      coverage mincov
      over     $py y (300:301)
      values   c[Long:"CRS:1"($px), Lat:"CRS:1"($py)]
   )
), "csv") 

comment:22 Changed 3 years ago by dmisev

There is a bug in the min itself:

rasql -q 'select csv(marray i in [-1:1] values condense min over j in [3:4] using i) from mr' --out string
  Result object 1: {0,0,0}

max works as expected:

rasql -q 'select csv(marray i in [-1:1] values condense max over j in [3:4] using i[0]) from mr' --out string
  Result object 1: {-1,0,1} 

comment:23 Changed 3 years ago by dmisev

Not even in the min:

rasql -q 'select 5 min 2 from mr' --out string
  Result element 1: 2

comment:24 Changed 3 years ago by dmisev

patch submitted.

comment:25 Changed 3 years ago by dmisev

  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.