NLnet; Reduce osm2pgsql resource usage (original) (raw)

More efficient database usage for OSM data

Osm2pgsql is used to import OpenStreetMap (OSM) data into a PostgreSQL/PostGIS database and keep it up to date. It is an essential tool of many map creation and OSM data analysis toolchains. It is used to serve millions of users daily on the OpenStreetMap project's own raster and vector map infrastructure. It is also a basis for the Nominatim geocoder. With the amount of data in OSM climbing continually, the memory and disk requirements of osm2pgsql have risen as well. In this project we want to reduce the memory and disk usage of osm2pgsql by implementing more efficient storage formats, specifically for "intermediate" data used while processing. This will not only help with resource consumption on the community run OSM servers, but also enable wider use of OSM data, even on planet-scale, in low-resource environments available to small NGOs or to students.