wiki:CommonFormatsInterface

Version 21 (modified by dmisev, 2 weeks ago) (diff)

--

Common formats interface

The encode/decode functions allow format parameters to be specified, in order to fine-tune the conversion process. Certain parameters are common to all formats:

  1. variable subset: variable names, band ids
  2. internal structure: lat/lon, x/y/t, message domains in GRIB
  3. x/y transpose: indicate if x/y should be transposed or is it not relevant (comes up in netCDF and GRIB and has a performance penalty at both decode and encode)
  4. filepath: absolute path to an input file, this improves ingestion performance if the data is on the same machine as the rasdaman server, as the network transport is bypassed
  5. subset domain: only the given subset needs to be extracted from the input file; note that the subset domain must match in dimensionality with the file dimensionality, and must be accordingly offset to the grid origin in the file, which is typically 0,0,0..

All parameters are optional.

decode

{
  // Specify variable names, band ids (0-based), etc.
  "variables": [ "var1", "var2", ... ],

  // Absolute path to input file(s). This improves ingestion performance if the data 
  // is on the same machine as the rasdaman server, as the network transport is bypassed;
  // It is possible that a format could have multiple files associated to each other,
  // so this argument is an array of filepaths.
  // Note: supported for netCDF, GRIB, and GDAL formats.
  "filePaths": [ "/path/to/file.tif", ... ],

  // Only the given subset needs to be extracted from the input file.
  "subsetDomain": "[0:100,0:100]",

  // Indicate if x/y should be transposed or is it not relevant (comes up in netCDF and GRIB
  // and has a performance penalty, so avoid if possible);
  // The argument is an array of 0-based axis ids indicating the axes that need to be transposed,
  // e.g. the first axis is 0, second is 1, etc; must be contiguous, [N,N+1]
  "transpose": [0,1],
 
  // Describe the internal structure of a format if necessary, e.g. message domains in GRIB
  "internalStructure": {
    "messageDomains": [
      { "msgId": 1, "domain": "[0:0,0:0,0:719,0:360]" },
      { "msgId": 2, "domain": "[0:0,1:1,0:719,0:360]" },
      { "msgId": 3, "domain": "[0:0,2:2,0:719,0:360]" },
      ..
    ]
  },

  // Extra format parameters
  "formatParameters": {
    "key": "value",
    ..
  },

  // Configuration options (string key/value pairs); details for GDAL: https://trac.osgeo.org/gdal/wiki/ConfigOptions
  "configOptions": {
    "GDAL_CACHEMAX": "64",
    ...
  }
}

encode

{
  // netCDF-specific
  "dimensions": [ "dimName1", "dimName2", ... ],

  // Specify variable names, band ids (0-based), etc; the variables can be objects listing details 
  // like type, metadata and data for netCDF, or simply an array of variable names
  "variables": {
    "var1": {..},
    "var2": {..},
    ... 
  },
 
  // single string, or multiple key/value pairs
  "metadata": {
    "standard_name" = "sea_surface_temperature",
    "long_name" = "Sea Surface Temperature",
    "units" = "K"
  },

  // Indicate if x/y should be transposed or is it not relevant (comes up in netCDF and GRIB and 
  // has a performance penalty, so avoid if possible);
  // The argument is an array of 0-based axis ids indicating the axes that need to be transposed,
  // e.g. the first axis is 0, second is 1, etc; must be contiguous, [N,N+1]
  "transpose": [0,1],

  // geo-referencing information, can be either given as a geo bounding box or an array of GCPs (but not both)
  // see 'Affine GeoTransform' and 'GCPs' sections at http://www.gdal.org/gdal_datamodel.html
  "geoReference": {
    "crs": "EPSG:4326",

    // a geo bounding box
    "bbox": {
      "xmin": -15,
      "ymin": 0.5, 
      "xmax": 10.2, 
      "ymax": 25
    },

    // or an array of GCPs
    "GCPs": [
      {
        "id": ..,     // optional unique identifier (gets the array index by default)
        "info": ..,   // optional text associated with the GCP

        // these must have double value
        "pixel": .., "line": ..,     // GCP location on the raster
        "x": ..,  "y": ..,  "z": ..  // georeferenced location, with "z" being optional (zero by default)
      },
      ...
    ]
  },

  // single value or an array of values if different for multiple bands
  // ( if nodata= 1 single value, it will be applied to all bands, and if nodata = 
  // array of values then each value is applied to each band separately )
  "nodata": 0,

  // for more details see "Color Table" at http://www.gdal.org/gdal_datamodel.html
  "colorPalette": {
    "paletteInterp": "RGB"       // optional palette interpretation, one of "Gray", "RGB", "CMYK", "HSL" (default: "RGB")
    // An optional array of band color interpretations, one of: Undefined Gray Palette Red Green Blue Alpha 
    // Hue Saturation Lightness Cyan Magenta Yellow Black YCbCr_Y YCbCr_Cb YCbCr_Cr YCbCr_Cr
    "colorInterp": [ "Red", "Green", "Blue" ]
    "colorTable": [              // optional color table, an array of arrays with 1, 3 or 4 short values
                                 // (depending on the colorInterpretation) for each color entry
      [255, 0, 0, 0],            // pixels with value 0 are colored red
      ...
    ]
  },

  // Format dependent extra parameters, e.g. gdal specific format parameters, compression, etc.
  "formatParameters": {
    "INTERLEAVE": "BAND",
    "COMPRESSION": "LZW",
    ...
  },

  // Configuration options (string key/value pairs); details for GDAL: https://trac.osgeo.org/gdal/wiki/ConfigOptions
  "configOptions": {
    "GDAL_CACHEMAX": "64",
    ...
  }
}

Example queries

  1. Using filePaths in JSON format (<[0:0] 1c> is the 1 dummy value)
    rasql --user rasadmin --passwd rasadmin -q 
         'UPDATE test_mr SET test_mr[0:255,0:210] ASSIGN shift(decode(<[0:0] 1c>, 
          "GDAL", "{\"filePaths\":[\"/home/rasdaman/mr_1.png\"]}"), [0,0]) WHERE oid(test_mr) = 6145'
    
  2. Transpose the last two axes of the output before encoding to PNG (WCPS):
    for c in (test_mr) return encode(c, "png", "{ \"transpose\": [0,1] }")