Scalable Geospatial (Vector) Data Science¶

Nikolai Janakiev @njanakiev ¶

Slides - janakiev.com/slides/scalable-geospatial-data-science
Repository - github.com/njanakiev/scalable-geospatial-data-science

(Geospatial) Big Data¶

OpenStreetMap¶

6,140,639,049 nodes
677,822,881 ways
7,940,285 relations

www.openstreetmap.org/stats/data_stats.html

PostgreSQL

PostgreSQL and PostGIS¶

PostgreSQL Parallelization¶

Since PostgreSQL 9.6 (2016)
Enabled by default since PostgreSQL 10 (2017)

PostGIS Parallelization¶

With tweaks since PostgreSQL 11 and PostGIS 2.5
Out of the box since PostgreSQL 12 and PostGIS 3

Scaling PostgreSQL¶

Columnar Storage (Foreign Data Wrapper)¶

cstore_fdw - citusdata.github.io/cstore_fdw/

GPU Processing¶

PG-Strom - heterodb.github.io/pg-strom

Distributed PostgreSQL¶

CitusDB - github.com/citusdata/citus
Postgres-XL - www.postgres-xl.org

Hadoop

Apache Hadoop¶

2003 Google - The Google File System

2004 Google - MapReduce: Simplified Data Processing on Large Clusters

2006 Apache Nutch - Hadoop

2007 Used by Facebook, LinkedIn, Twitter, among others

2008 Apache Foundation Project

Apache Hadoop¶

Hadoop Distributed File System (HDFS)¶

2003 Google - The Google File System

Hadoop MapReduce¶

2004 Google - MapReduce: Simplified Data Processing on Large Clusters

Hadoop YARN¶

Apache Spark¶

Created 2009, open sourced 2010

2010 - Spark: Cluster Computing with Working Sets

distributed computation engine, written in Scala, with bindings for Java, Python, and R

GeoMesa (geomesa.org)¶

LocationTech under the Eclipse Foundation

Spatial Indexing¶

Space Filling Curve (2d/3d) - Index Overview

Database¶

Integrations¶

Apache Spark RDD/SQL Interface
GeoServer Integration

Beyond the Elephants¶

Presto (prestodb.io) or Trino (trino.io)¶

Distributed SQL Query Engine for Big Data

Presto Connectors - connectors to most common SQL and NoSQL databases

Geospatial Functions - various common ST_ functions available, uses RTree indexing Issue #13079 for spatial joins

Dask (dask.org)¶

Parallel and distributed computing library for analytics written in Python

GeoPandas (geopandas.org)¶

Extends the datatypes used by Pandas to allow spatial operations on geometric types

rtree - Spatial index for Python GIS

(Experimental) geopandas/dask-geopandas - New implementation of GeoPandas and Dask

ElasticSearch (elastic.co)¶

Geo Point Indexing ¶

GeohashPrefixTree
QuadPrefixTree
BKD-trees (since ES 6.0) - BKD-backed geo_shapes in Elasticsearch: precision + efficiency + speed (2019)

Geo queries ¶

geo_bounding_box query
geo_distance query
geo_polygon query
geo_shape query

MongoDB (mongodb.com)¶

Document-oriented NoSQL database which uses JSON-like documents with optional schemas

Geospatial Index¶

GeoJSON Object
2d Index, 2d Index Internals - Uses GeoHash Indexing
2d Sphere Index
geoHaystack Indexes - Special index that is optimized to return results over small areas

Geospatial Queries ¶

$geoIntersects
$geoWithin
$near
$nearSphere
$geoNear

Redis (redis.io)¶

in-memory data structure store, used as a database, cache and message broker

Geospatial Indexing¶

Geohash Indexing

Geospatial Commands ¶

Scalable Geospatial Data Science¶

Nikolai Janakiev @njanakiev ¶

Slides - janakiev.com/slides/scalable-geospatial-data-science
Repository - github.com/njanakiev/scalable-geospatial-data-science

Resources¶