osm2pgsql icon indicating copy to clipboard operation
osm2pgsql copied to clipboard

Allow flex backend to specify additional indexes to create

Open pnorman opened this issue 3 years ago • 1 comments

Many transforms need additional indexes to be used. Currently this is commonly done as a .sql file distributed alongside the transforms, but this is not ideal for a few reasons

  1. It is an extra process that might be forgotten
  2. The create index statements will need to vary with the FILLFACTOR depending on if the database is updatable or not
  3. The naïve way of running osm2pgsql && psql -f indexes.sql will lead to a period of time when only one index is being built. This is especially bad for slim imports with the way node index.
  4. Index builds should be run in parallel, but psql -f can't handle this, requiring more elaborate SQL runners
  5. VACUUM ANALYZE should be run after index building, not before

Instead, the index builds should be run, in parallel, for each table once the table has been sorted

With the flex backend it should be possible to specify multiple indexes for each table. It needs to be possible to specify the name (e.g. admin), index type (e.g. GIST), columns or expression (e.g. ST_PointOnSurface(way)), and partial index predicate (e.g. name IS NOT NULL AND boundary = 'administrative' AND admin_level IN ('0', '1', '2', '3', '4')). osm2pgsql would supply the storage_parameter based on if it is updatable and tablespace_name, if a tablespace is set.

pnorman avatar Jul 09 '22 22:07 pnorman

FYI: we've been discussing this use case as part of a more generic post processing some time ago: https://github.com/openstreetmap/osm2pgsql/issues/1211#issuecomment-645287526

mmd-osm avatar Jul 10 '22 08:07 mmd-osm

It is now possible to create any kind of indexes you want with the flex output. See https://osm2pgsql.org/doc/manual.html#defining-indexes for details.

joto avatar Dec 07 '22 19:12 joto