openstreetmap-cgimap

Introduction

openstreetmap-cgimap is a C++ application designed to improve the performance of key parts of the OpenStreetMap (OSM) API. It directly interacts with the PostgreSQL backend (APIDB) and bypasses the Rails-based OSM website.

This tool is useful for replacing certain API endpoints with optimized versions to enhance performance. For more details, refer to the cgimap wiki page wiki page.

Key Features

• Optimized API Endpoints: cgimap provides 25 performance-enhanced endpoints for interacting with OSM data, including fetching nodes, ways, relations, uploading changesets, and more.

• Efficient Interaction with PostgreSQL Database: It connects directly to the PostgreSQL-backed OSM database (APIDB) without requiring the OSM website application.

• Bulk enabled database operations: Mass database operations avoid unnecessary roundtrips for improved performance.

Prerequisites

Before installing and using cgimap, make sure your system meets the following requirements: Software Requirements:

• Operating System: Linux (tested on Ubuntu 22.04, 24.04, Debian 12, and 13)

• C++20: cgimap requires a compiler that supports C++20.

• PostgreSQL: A running PostgreSQL server with an APIDB backend. Follow the Rails website instructions to set up an APIDB backend.

• Dependencies: Install the following packages on Ubuntu/Debian:

    sudo apt-get install libxml2-dev libpqxx-dev libfcgi-dev zlib1g-dev libbrotli-dev \
         libboost-program-options-dev libfmt-dev libmemcached-dev libyajl-dev

Note that the full set of packages needed from a fresh install (tested with Ubuntu 22.04, Debian 12 and 13) - you may already have many or all of these - is:

    sudo apt-get install git build-essential cmake make

For Ubuntu 24.04 users, you might need to build and install libpqxx version 7.10.0 manually due to compatibility issues with C++20. More details can be found in docker/ubuntu/Dockerfile2404.

Installation

Clone the Repository

First, clone the cgimap repository to your local machine:

    git clone https://github.com/zerebubuth/openstreetmap-cgimap.git
    cd openstreetmap-cgimap/

Build openstreetmap-cgimap

Create a build directory and use cmake to compile the program:

    mkdir build
    cd build
    cmake ..
    cmake --build .

After the build is complete, you'll have a openstreetmap-cgimap executable.

Install openstreetmap-cgimap (Optional)

sudo make install

Configuration

Running openstreetmap-cgimap

Once cgimap is built, you can run it with the following command (adjust parameters as needed):

    ./openstreetmap-cgimap --dbname=openstreetmap --username=user --password=pass \
         --dbport=5432 --socket=:54321 --logfile=/tmp/logfile \
         --daemon --instances=10

You can limit access to the FastCGI server by specifying a local IP address or using Unix domain sockets.

Environment Variables and Config File

You can configure cgimap using environment variables or an INI-style config file. For example, to set the database parameters, add them to your cgimap.config file:

dbname=openstreetmap
host=localhost
username=user
password=pass

#update-host=127.0.0.1
#update-dbname=openstreetmap
#ratelimit=100000

# Expert settings (should be left to their default values in most cases)
# see --help for further details
#
#map-area=10
#disable-api-write=
#max-payload=50000000

Typically you will need to modify the database connection parameters and path to the executable. See ./openstreetmap-cgimap --help for a list of options. Besides, rate limiting parameters need to be configured according to your needs.

To convert a command line option to an environment variable prepend CGIMAP_ to the option, convert hyphens to underscores, and capatalize it. For example, the option --moderator-ratelimit becomes the environment variable CGIMAP_MODERATOR_RATELIMIT.

Run cgimap with the config file:

./openstreetmap-cgimap --configfile=/path/to/cgimap.config

Daemon Mode

To run cgimap as a background service, you can use systemd (for modern systems). See the relevant scripts in the scripts directory.

For systemd:

sudo cp scripts/cgimap.service /etc/systemd/system/cgimap.service
sudo systemctl enable cgimap
sudo systemctl start cgimap

An example of this can be found in OSM Chef.

Database Permissions

Ensure the database user has proper permissions to read and update APIDB tables. Refer to the OSM chef repo for a list of required permissions for cgimap. We recommended to use a dedicated database user for cgimap, as well as a dedicated OS user to run cgimap.

Use with a Web Server

cgimap has to be used with a FastCGI enabled HTTP server like lighttpd, apache2 etc. See the instructions below to use cgimap with lighttpd or apache2.

Configuring lighttpd as FastCGI proxy

A sample lighttd.conf file is provided, which can be used for testing purposes only. To test cgimap with lighttpd you will need to install lighttpd:

sudo apt-get install lighttpd

Edit the supplied lighttpd.config file to include your cgimap path and run it with the lighttpd like

/usr/sbin/lighttpd -f lighttpd.conf

You can then access the running instance at http://localhost:31337/api/0.6/map?bbox=...

Configuring Apache as FastCGI proxy

Fcgi programs can be deployed with Apache using mod_fastcgi_handler, mod_fcgid, mod_fastcgi, and on recent versions mod_proxy_fcgi.

A sample Apache configuration file that will work in conjunction with cgimap as a daemon is supplied in scripts/cgimap.conf. To use this on a Ubuntu-based system you need to copy the configuration to where Apache will read it and create an api directory:

    sudo cp scripts/cgimap.conf /etc/apache2/sites-available/cgimap
    sudo chmod 644 /etc/apache2/sites-available/cgimap
    sudo chown root:root /etc/apache2/sites-available/cgimap
    sudo mkdir /var/www/api
    sudo a2ensite cgimap
    sudo service apache2 restart

The apache modules mod_proxy and mod_fastcgi_handler must also be enabled.

By the way, the OSMF api.openstreetmap.org instance runs cgimap as a daemon and Apache with mod_proxy_fcgi.

Docker Support

You can build a Docker image using one of the provided Dockerfiles underneath docker/

docker build -f docker/debian/Dockerfile_bookwork . -t cgimap:bookworm

Once the image is built, you can run cgimap in a container. For more details, refer to the Docker documentation.

zerebubuth#213 has additional configuration details on how to use the cgimap image in a complete development environment, which includes the Rails port, a PostgreSQL DB, lighttpd as reverse proxy, and openstreetmap-cgimap.

Visual Studio DevContainer Support

If you are using Visual Studio Code, you can set up a DevContainer to develop, build, test and run cgimap in an isolated development environment without worrying about your local dependencies.

How to Use:

• Install VS Code: If you haven't already, install Visual Studio Code.

• Install Docker: Make sure Docker is installed and running on your machine, as DevContainers rely on Docker.

• Open the Repository in VS Code:

  • Open the cgimap repository in VS Code.
  • If prompted, open the project in a DevContainer.
  • Alternatively, use the Remote - Containers extension to reopen the project inside the DevContainer.

• Build and Run cgimap: Once inside the DevContainer, you can follow the usual build steps:

    mkdir build
    cd build
    cmake ..
    cmake --build .

This setup running on Debian Bookworm (like the official OSMF instance) ensures that you have all necessary dependencies inside the container. You can also build and deploy Debian packages using dpkg-buildpackage inside the DevContainer.

Developing on WSL2 is also possible. Start vscode inside WSL2, and continue with opening the repository as described earlier.

Running Tests

If you'd like to run the test suite, install the required packages:

sudo apt-get install postgresql-all postgresql-common postgresql-server-dev-all

Then, configure the build to include tests:

cmake .. -DCMAKE_BUILD_TYPE=Debug

or

cmake .. -DBUILD_TESTING=ON

Run the tests:

ctest

Test cases are executed under pg_virtualenv, i.e. no further steps are needed to create databases as your user.

To speed up test runs, you can use cmake option -DENABLE_PGVIRTUALENV=OFF, then run the tests using:

pg_virtualenv ctest

This way, the virtual PostgreSQL cluster can be reused across test cases.

Acknowledgements

cgimap contains code from and is partly based on the following:

• quad_tile.c by TomH.
• See contrib directory for more details on:
  • libxml++ by The libxml++ development team
  • Catch2 by Catch2 Authors
  • SJParser by Denis Tikhomirov