#1237 closed defect (fixed)

Rasql_Auto round floating point value can make wrong result

Reported by: bphamhuu Owned by: rdeliallisi
Priority: major Milestone: 10.0
Component: conversion Version: development
Keywords: WCPS 2.0, round floating points, wrong results Cc: dmisev
Complexity: Medium

Description

WCPS query

for c in (mr) return encode(  pow( c[i(100:110),j(100:110)], 0.5)  + 200000 , "csv" )

will returns {200013, 200014, 200015,...}

but it should returns the correct floating points.

for c in (mr) return encode(  pow( c[i(100:110),j(100:110)], 0.5)  , "csv" )

which returns

{14.3178,14.2127,14.0712,13.9284,13.8924,13.8564,13.7477,13.7477,13.8203,13.8924,13.8203}

then when + 200000 it should be {200014.3178, 2000014.2127, 2000013.8924,...}

The problem is with this behavior, when using statistic the data is "fixed" and can result untrusted result, e.g counting point values > 2000014.200 (suppose has 100 points) but because 14.200 is converted to 14 (and + 200000 to 200014) then only returns 10 points.

Change History (8)

comment:1 Changed 14 months ago by bphamhuu

  • Component changed from undecided to rasql

comment:2 Changed 14 months ago by dmisev

Why rasql, have you tested if the issue is in rasql actually? What is the rasql query?

comment:3 Changed 14 months ago by bphamhuu

Dimitar, Rasql query

select csv(((pow(c[100:110,100:110], 0.5))+(200000))) from mr AS c where oid(c)=45569

And tested

rasql -q "select csv(((pow(c[100:110,100:110], 0.5))+(200000))) from mr AS c where oid(c)=45569" --out string
rasql: rasdaman query tool v1.0, rasdaman v9.2.0-beta1-gf79e291 -- generated on 21.01.2016 14:39:26.
opening database RASBASE at localhost:7001...ok
Executing retrieval query...ok
Query result collection has 1 element(s):
  Result object 1: 

{200013,200013,200013,200013,200013,200013,200013,200013,200013,200014,200014},...
rasql done.

comment:4 Changed 14 months ago by dmisev

It seems like a bug in the csv function in particular, exporting to tiff or netcdf for example gives the right values. min_cells/max_cells also gives wrong values.

comment:5 Changed 14 months ago by dmisev

  • Component changed from rasql to conversion

comment:6 Changed 14 months ago by bphamhuu

thanks for updating :)

comment:7 Changed 14 months ago by dmisev

  • Owner set to rdeliallisi
  • Status changed from new to assigned
rasql -q 'select netcdf(((pow(c[100:110,100:110], 0.5))+(200000.0))) from test_grey2 AS c' --out file
ncdump rasql_1.nc

This will show correct values. Probably it's some precision issue with C++ streams which csv uses.

comment:8 Changed 11 months ago by dmisev

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.