wiki:Features

Version 42 (modified by pbaumann, 4 years ago) (diff)

--

Features

This is a high-level feature summary of rasdaman. For details refer to Why rasdaman??, Technology?, the documentation, and the scientific publications of the L-SIS research group maintaining rasdaman.

Per se, rasdaman is domain neutral, which makes it suitable for all applications where raster data management is an issue.

The petascope package of rasdaman (see documentation) is a servlet package implementing OGC standard interfaces WCS, WCPS, WCS-T, and WPS; see EarthLook for a kaleidoscope of hands-on interactive demos.

Features described on this page below make up the rasdaman community version. Add-ons with performance boosters (such as just-in-time compilation and tape silo support) are available with rasdaman enterprise from rasdaman GmbH. The feature table below summarizes the differences.

Information on this page is voluntary and subject to errors and unannounced modifications. No rights, in particular against rasdaman GmbH, can be construed whatsoever from any information on this website.

Summary: rasdaman community versus rasdaman enterprise

The following table shows feature availability in the open source version, rasdaman community, and the commercial variant, rasdaman enterprise.

Note that both rasdaman versions are complete, fully operational systems, without reservation. The rasdaman enterprise version mainly adds performance boosters.

rasdaman version: communityenterprise
Main functionality:
rasql XXevery community query will run on both versions
C++ API XXC++ clients using community functionality will run on both versions
Java API XXJava clients using community functionality will run on both versions
Extra functionality:
dynamic data definition XXnew cell, array, and collection types can be added at runtime
null values X
CRS transformation X
Storage:
tiling XX
tiling policies XX
storage layout language XX
spatial indexes XX
input/output data formatsXX
compression X
tape archive integration X
Performance accelerators:
parallel server processesXX
distributed processing X
heuristic optimization X
multi-threaded server X
just-in-time compilation X
GPU query evaluation X
server management:
via command line XX
via Web browser X
OGC interfaces:
WMS 1.1 X thanks to volunteers from the community porting the code from enterprise (provided by rasdaman GmbH) to community
WMS client X...but any other (such as OpenLayers) will do as well, of course
WCS Core XXWCS 2.0.1 based on GMLCOV 1.0.1 (both OGC's current version) and WCS 1.1 (deprecated)
WCS Encoding Extension: GeoTIFF 0.x.x XXIn the OGC adoption process; under implementation in rasdaman; note that all GDAL-supported formats are already available in rasdaman
WCS Encoding Extension: JPEG2000 0.x.x XXSame as above; also includes GMLJP2
WCS Service Extension: Range Subsetting 1.0.1 XX
WCS Service Extension: Transaction 1.1.4 XX
WCS Service Extension: Processing 1.0.0 XX This is the adapter linking WCPS into WCS
WCS Service Extension: Scaling 0.x.x XX In the OGC adoption process
WCS Service Extension: Interpolation 0.x.x XX In the OGC adoption process
WCS Service Extension: CRS 0.x.x XIn the OGC adoption process
WCS Protocol Extension: GET/KVP 1.0.1 XX
WCS Protocol Extension: POST/XML 1.0.0 XX
WCS Protocol Extension: SOAP 1.0.0 XX
WCS Protocol Extension: REST 0.x.x XXIn the OGC adoption process
WCPS 1.0 XX
WPS 1.0 XX
Base DBMSs supported:
PostgreSQL XX
MySQL X
Oracle X
IBM Informix X
IBM DB2 X
file system X

Array data model

Arrays are determined by their extent ("domain") and their cell ("pixel", "voxel"). The rasdl type definition language (aka DDL) allows to dynamically define new types; syntax is kept close to ODMG's ODL. All base and composite data types allowed in languages like C/C++ (except for pointers and arrays) can be defined as cell types, including nested structs. For example, the following stanza defines an XGA image:

typedef marray< [0:1023,0:767], struct{unsigned char red, green, blue;} > XGA;

Over such typed arrays, collections (ie, tables - ODMG style, again) are built. Collections have two columns (attributes), a system-maintained object identifier (OID) and the array itself. This allows to conveniently embed arrays into relational modeling: foreign keys in conventional tables allow to reference particular array objects, in connection with a domain specification even parts of arrays. A collection of XGA images is defined in rasdl through

typedef set< XGA > XgaSet;

This collection type can be instantiated through

create collection XgaSet myXgaCollection; 

After this instantiation, the collection is ready for population through rasql.

Query language

The rasdaman query language, rasql, offers raster processing formulated through expressions over raster operations in the style of SQL. Consider the following query: "The difference of red and green channel from all images from collection LandsatImages where somewhere in the red channel intensity exceeds 127". In rasql, it is expressed as

select ls.red - ls.green
from LandsatImages as ls
where max_cells( ls.red ) > 127

Rasql is a full query language, supporting select, insert, update, and delete. Additionally, the concept of a partial update is introduced which allows to selectively update parts of an array. In view of the potentially large size of arrays this is a practically very relevant feature, e.g., for updating satellite image maps with new incoming imagery.

Query formulation is done in a declarative style (queries express what the result should look like, not how to compute it). This allows for extensive optimization on server side. Further, rasql is safe in evaluation: every valid query is guaranteed to to terminate in finite time.

C++ and Java API

Client development is supported by the C++ API, raslib, and the Java API, rasj; both adhere to the ODMG standard. Communication with a rasdaman database is simple: open a connection, send the query string, receive the result set. Iterators allow convenient acecss to query results.

Once installed, go into the share/rasdaman/examples subdirectory to find sample code.

Tiled storage

On server side, arrays are stored inside a standard database. To this end, arrays are partitioned into subarrays called tiles; each such tile goes into a BLOB (binary large object) in a relational table. This allows conventional relational database systems to maintain arrays of unlimited size.

A spatial index allows to quickly locate the tiles required for determining the tile set addressed by a query.

The partitioning scheme is open - any kind of tiling can be specified during array instantiation. A set of tiling strategies is provided to ease administrators in picking the most efficient tiling.

Tile streaming

Query evaluation in the server follows the principle of tile streaming. Each operator node processes a set of incoming tiles and generates an output tile stream itself. In many cases this allows to keep only one database tile at a time in main memory. Query processing becomes very efficient even on low-end server machines.

Server multiplexing

A rasdaman server installation can consist of an arbitrary number of rasdaman server processes. A dynamic scheduler, rasmgr, receives incoming connection requests and assigns a free server process. This server process then is dedicated to the particular client until the connection is closed. This allows for highly concurrent access and, at the same time, increases overall safety as clients are isolated against each other.

OGC geo service standards support

While rasdaman itself is domain agnostic and supports any array application, the petascope servlet, as part of rasdaman, adds in geo semantics, such as dealing with geo coordinates. To this end, rasdaman implements the Open Geospatial Consortium standards for gridded coverages, i.e., multi-dimensional raster data. The OGC service interfaces supported are

The Princial Architect of rasdaman, Peter Baumann, is chair of the OGC WCS Standards Working Group (WCS.SWG) and editor of coverage model (GMLCOV), WCPS, and most of the WCS specifications, rasdaman naturally has become Reference Implementation for several of these standards and usually implements them first and way ahead of other systems, even before final adoption. Likewise, any changes to coverage-related specifications usually are verified in rasdaman first and, hence, become available early. The same holds for the OGC conformance testing of coverage services where rasdaman code contributors have a lead. In summary, rasdaman can be considered the most comprehensive and best tested implementation of the OGC coverage standards.