Opened 5 months ago

Closed 5 months ago

#1492 closed enhancement (fixed)

Petascope_UploadFileHandler from WCST_import uses filePaths in rasql query

Reported by: dmisev Owned by: bphamhuu
Priority: critical Milestone: 9.4
Component: wcst_import Version: development
Keywords: Cc: vmerticariu, bphamhuu, pbaumann
Complexity: Medium

Description (last modified by bphamhuu)

The format parameters of decode allow specifying filepaths as follows:

  // Absolute path to input file(s). This improves ingestion performance if the data 
  // is on the same machine as the rasdaman server, as the network transport is bypassed;
  // It is possible that a format could have multiple files associated to each other,
  // so this argument is an array of filepaths.
  // Note: supported for netCDF, GRIB, and GDAL formats.
  "filePaths": [ "/path/to/file.tif", ... ],

It would be good to make use of this option in wcst_import when the conditions are right (data on same machine as rasserver): we'll save quite some data reading / copying / writing.

Example query:

rasql --user rasadmin --passwd rasadmin -q 
     'UPDATE test_mr SET test_mr[0:255,0:210] ASSIGN shift(decode(<[0:0] 1c>, 
      "GDAL", "{\"filePaths\":[\"/home/rasdaman/mr_1.png\"]}"), [0,0]) WHERE oid(test_mr) = 6145'

Change History (11)

comment:1 Changed 5 months ago by dmisev

I think this should be default behaviour actually, it's very rare to ingest data to a remote server? wcst_import can't even ingest to a remote server if I'm not wrong?

comment:2 Changed 5 months ago by vmerticariu

The blocker is in petascope, but you're right, currently importing is only supported locally.

comment:3 Changed 5 months ago by dmisev

  • Milestone changed from Future to 9.4
  • Owner set to bphamhuu
  • Status changed from new to assigned

Ok, so we should just use the filePaths when WCST generates the decode format parameters.

comment:4 Changed 5 months ago by dmisev

  • Priority changed from major to critical

Raising priority, it's a simple fix but will have quite some impact.

comment:5 Changed 5 months ago by bphamhuu

I think by default, it already used this feature as no file was uploaded to a temporary folder and then running rasql update from this folder, have a look at a updated slice from UpdateCoverageHandler?.

rasql --user rasadmin --passwd rasadmin -q 'UPDATE test_ansidate_different_crs_origin_5 SET test_ansidate_different_crs_origin_5[0:0,0:4318,0:8640] ASSIGN shift(decode($1, "NetCDF", "{\"variables\": [\"MERIS_nobs_sum\", \"chlor_a\"]}"), [0,0,0]) WHERE oid(test_ansidate_different_crs_origin_5) = 1025'

--file '/home/rasdaman/trask/pixel_shiftted/Chloro/ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6-201003-fv2.0.nc

comment:6 Changed 5 months ago by dmisev

You'd still run rasql but without the --file option now. Instead the file is specified in the format parameters like this:

rasql --user rasadmin --passwd rasadmin -q 'UPDATE test_ansidate_different_crs_origin_5 SET test_ansidate_different_crs_origin_5[0:0,0:4318,0:8640] ASSIGN shift(decode($1, "NetCDF", "{\"variables\": [\"MERIS_nobs_sum\", \"chlor_a\"], \"filePaths\": [ \"/home/rasdaman/trask/pixel_shiftted/Chloro/ESACCI-OC-L3S-CHLOR_A-MERGED-1M_MONTHLY_4km_GEO_PML_OC4v6-201003-fv2.0.nc\" ]}"), [0,0,0]) WHERE oid(test_ansidate_different_crs_origin_5) = 1025'

comment:7 Changed 5 months ago by bphamhuu

  • Resolution set to invalid
  • Status changed from assigned to closed

comment:8 Changed 5 months ago by bphamhuu

  • Resolution invalid deleted
  • Status changed from closed to reopened

comment:9 Changed 5 months ago by bphamhuu

sorry, I could not see Dimitar's comment as email was not fowarded, then closed the ticket.

comment:10 Changed 5 months ago by bphamhuu

  • Description modified (diff)
  • Summary changed from Support filePaths in wcst_import to Petascope_UploadFileHandler from WCST_import uses filePaths in rasql query

comment:11 Changed 5 months ago by bphamhuu

  • Resolution set to fixed
  • Status changed from reopened to closed

patch was accepted, close ticket.

Note: See TracTickets for help on using tickets.