Data Science with OpenStreetMap and Wikidata

Nikolai Janakiev @njanakiev

Outline

Part I: Wikidata and OpenStreetMap

• Difference between Wikidata and OpenStreetMap
• Ways to connect data between Wikidata and OpenStreetMap

Part II: Data Science with Wikidata and OSM

• Libraries and Tools
• Exhibition of Various Analyses and Results

OpenStreetMap Elements

Metadata in OpenStreetMap

Wikidata is a Knowledge Graph

Querying Wikidata with SPARQL

• https://query.wikidata.org/

All Windmills in Wikidata

SELECT ?item ?itemLabel ?image ?location ?country ?countryLabel
WHERE {
  ?item wdt:P31 wd:Q38720.
  OPTIONAL { ?item wdt:P18 ?image. }
  OPTIONAL { ?item wdt:P625 ?location. }
  OPTIONAL { ?item wdt:P17 ?country. }
  SERVICE wikibase:label { 
    bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". 
  }
}

Query link

OpenStreetMap and Wikidata in Numbers

OpenStreetMap (Source)

• Started 2004
• Number of users: 5,630,923
• Number of uploaded GPS points: 7,459,170,764
• Number of nodes: 5,424,072,098
• Number of ways: 601,538,972
• Number of relations: 7,038,670

Wikidata (Source)

• Started 2012
• Number of active users: 20,798
• Number of items: 59,218,423
• Number of edits: 1,000,545,117

Linking OpenStreetMap with Wikidata?

File:WdOsm-semanticBridge.jpg

OpenStreetMap to Wikidata

• wikidata=* tag (stable)

Wikidata to OpenStreetMap

• OSM relation ID (P402), in total 97704 entities (unstable)
  Note: Should not be used for Nodes, Ways or Areas

• Permanent ID proposal

• OSM tag or key (P1282) mapping of OSM key-values to Wikidata entities, in total 1862 entities (e.g. lighthouse and Tag:man_made=lighthouse)

Data Science

• Donoho, David. "50 years of data science." Journal of Computational and Graphical Statistics 26.4 (2017): 745-766.

Used Tools and Libraries

• Jupyter - interactive notebook development environment
• PostGIS - spatial database extender for PostgreSQL
• GDAL ogr2ogr - converting simple features between file formats

Python Libraries

• NumPy - numerical and scientific computing
• Pandas - data analysis library
• Matplotlib - 2D plotting library
• Shapely - analysis and manipulation of GEOS features
• GeoPandas - Pandas extension for spatial operations and geometric types
• PySAL - spatial analysis library
• Datashader - graphics pipeline system for large datasets

OpenStreetMap Elements with Wikidata Tag

Wikidata Instance of (P31) Property

Analyzing Websites Regionally

I like how everyone is saying that jQuery is dead and at the same time - it powers 70% of the Web

— Tomasz Łakomy (@tlakomy) 19 June 2019

Classifying Countries and Regions with OSM

• Using counts of various amenities as signatures for regions
• osm.janakiev.com

Castle Dossier Map of Switzerland

• Thematic maps made with OpenStreetMap, Wikidata, Wikimedia Commons (images) and Wikipedia castle-map.infs.ch, Burgen-Dossier_Schweiz - OSM Wiki
• BLICK sucht die schönsten Schlösser und Burgen der Schweiz

Conclusion

• Naming things is hard, meaningfully categorizing even harder

• Wikidata can tends to show variations in definitions between countries but tends to be consistent within countries (this hypothesis has not been tested)

• Wittgensteins ruler: "When you use a ruler to measure the table, you are also using the table to measure the ruler.", Biased data can tell you more about the people behind the data than the data itself

• Know Thy Data: Data Provenance and Completeness is crucial for data analysis and prediction

Data Completeness

OpenStreetMap

• Barrington-Leigh, Christopher, and Adam Millard-Ball. "The world’s user-generated road map is more than 80% complete." PloS one 12.8 (2017): e0180698.
• wiki.openstreetmap.org/wiki/Completeness

Wikidata

• Michael Luggen, Djellel Difallah, Cristina Sarasua, Demartini and Philippe Cudré-Mauroux. "How to estimate completeness of classes in Wikidata." Sociebyte. 2019.
• Ahmeti, Albin, Simon Razniewski, and Axel Polleres. "Assessing the completeness of entities in knowledge bases." European Semantic Web Conference. Springer, Cham, 2017.
• COOL-WD: A Completeness Tool for Wikidata

Data Science with OpenStreetMap and Wikidata

Nikolai Janakiev @njanakiev

• Slides @ https://janakiev.com/slides/data-science-osm-wikidata

Resources

• Wikidata - OpenStreetMap Wiki
• FOSSGIS 2016: OpenStreetMap und Wikidata - Michael Maier
• FOSDEM 2019: Linking OpenStreetMap and Wikidata A semi-automated, user-assisted editing tool - Edward Betts
• WDTools - Wikidata Utilities and Tools