Skip to content

Commit 7584fc4

Browse files
Rub21Rub21
authored
Updating data process pipeline for Imposm - Vtiles (#671)
* Remove generalized_tables for admin adn create from single table the mviews * Update mviews refresh * Restore tiler db * Update deprecated imposm tags pseudoarea and exclude_tags * Update reject section for imposm * Update admin centroids * Build imposm image for staging * Standardize transport_lines and route layers using the same simplification value * Refactor zoom lavel for transport line and routes * Remove generalized_tables for water and udpate mviews * Standardize zoom level - water and landuse * Standardize zoom level - landuse and transport lines * Standardize zoom level - other areas , routes and transport areas * Comment refresh functions * Update toml file for vtiles and create documentation * Fix discrepancies mviews adn tegola config * Create mviews from high zoom level views * Update refresh views script * Add condition to run again the script * Delete views in cascade in routes * Fix missing semicoloe in transport lines * Update water areas layer * Update landuse lines layer - filter only tree_row * Refresh landuse lines mviews * Update queries form water areas, and landuse * Simplify materialized views by deriving them from higher-level materialized views * Update id_fieldname for admin lines * Update doc and transport areas * Add filters for each mview in docs * Update docker images got vtiles production
1 parent 9ffbdea commit 7584fc4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+1651
-1785
lines changed

‎.github/workflows/chartpress.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ on:
55
- 'main'
66
- 'staging'
77
- 'staging_deploy'
8-
- 'tiler_refresh'
8+
- 'vtiles_admin'
99
jobs:
1010
build:
1111
runs-on: ubuntu-22.04

‎DOC-VTILES.md‎

Lines changed: 260 additions & 0 deletions
Large diffs are not rendered by default.

‎hetzner/tiler/tiler.production.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ services:
2626
# imposm3
2727
imposm_production:
2828
container_name: tiler_imposm_production
29-
image: ghcr.io/openhistoricalmap/tiler-imposm:0.0.1-0.dev.git.3183.h59d5a56
29+
image: ghcr.io/openhistoricalmap/tiler-imposm:0.0.1-0.dev.git.3226.h9da5253
3030
volumes:
3131
- tiler_production_imposmdata:/mnt/data
3232
command:
@@ -49,7 +49,7 @@ services:
4949
# Tiler server
5050
tiler_server_production:
5151
container_name: tiler_server_production
52-
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3175.hbdd6021
52+
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3225.h2122f3e
5353
ports:
5454
- "9090:9090"
5555
env_file:
@@ -91,7 +91,7 @@ services:
9191

9292
tile_global_seeding_production:
9393
container_name: tiler_global_seeding_production
94-
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3175.hbdd6021
94+
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3225.h2122f3e
9595
env_file:
9696
- .env.production
9797
volumes:
@@ -111,7 +111,7 @@ services:
111111

112112
tile_coverage_seeding_production:
113113
container_name: tiler_coverage_seeding_production
114-
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3175.hbdd6021
114+
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3225.h2122f3e
115115
env_file:
116116
- .env.production
117117
volumes:
@@ -148,10 +148,10 @@ services:
148148
volumes:
149149
tiler_production_pgdata:
150150
driver: local
151-
name: tiler_db_18122025
151+
name: tiler_db_0801_v1
152152
tiler_production_imposmdata:
153153
driver: local
154-
name: tiler_imposm_18122025
154+
name: tiler_imposm_0801_v1
155155

156156
networks:
157157
ohm_network:

‎hetzner/tiler/tiler.staging.yml‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,11 @@ services:
2121

2222
imposm_staging:
2323
container_name: tiler_imposm_staging
24-
image: ghcr.io/openhistoricalmap/tiler-imposm:0.0.1-0.dev.git.3183.h59d5a56
24+
image: ghcr.io/openhistoricalmap/tiler-imposm:0.0.1-0.dev.git.3226.h9da5253
25+
# image: tiler-imposm:staging
26+
# build:
27+
# context: ../../images/tiler-imposm
28+
# dockerfile: Dockerfile
2529
volumes:
2630
- tiler_staging_imposmdata:/mnt/data
2731
command:
@@ -35,10 +39,13 @@ services:
3539
networks:
3640
- ohm_network
3741

38-
3942
tiler_server_staging:
4043
container_name: tiler_server_staging
41-
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3175.hbdd6021
44+
image: ghcr.io/openhistoricalmap/tiler-server:0.0.1-0.dev.git.3225.h2122f3e
45+
# image: tiler-server:staging
46+
# build:
47+
# context: ../../images/tiler-server
48+
# dockerfile: Dockerfile
4249
ports:
4350
- "9091:9090"
4451
env_file:
@@ -106,10 +113,10 @@ services:
106113
volumes:
107114
tiler_staging_pgdata:
108115
driver: local
109-
name: tiler_db_1212_v2
116+
name: tiler_db_05_01_v9
110117
tiler_staging_imposmdata:
111118
driver: local
112-
name: tiler_imposm_1212_v2
119+
name: tiler_imposm_05_01_v9
113120

114121
networks:
115122
ohm_network:

‎images/tiler-db/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@ RUN git clone https://github.com/OpenHistoricalMap/DateFunctions-plpgsql.git .
1515

1616
# Deploy improvements to negative year handling
1717
RUN git checkout d9f06a4ac0674881247c38b29ba58ff1d167b7bb
18-
RUN cp datefunctions.sql /docker-entrypoint-initdb.d/datefunctions.sql
18+
RUN cp datefunctions.sql /docker-entrypoint-initdb.d/datefunctions.sql

‎images/tiler-db/scripts/docker-entrypoint.sh‎

100755100644
File mode changed.

‎images/tiler-db/scripts/update_db.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,4 @@ for DB in template_postgis "$POSTGRES_DB" "${@}"; do
77
psql --dbname="$DB" -c "
88
CREATE EXTENSION IF NOT EXISTS hstore;
99
"
10-
done
10+
done

‎images/tiler-imposm/Dockerfile‎

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM osgeo/gdal:ubuntu-small-3.2.3
1+
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.9.3
22

33
RUN apt-get -y update && apt-get install -y \
44
g++ \
@@ -18,21 +18,22 @@ RUN apt-get -y update && apt-get install -y \
1818
curl \
1919
wget \
2020
unzip \
21-
software-properties-common && \
21+
ca-certificates \
22+
software-properties-common \
23+
python3-pip \
24+
postgresql-client && \
2225
rm -rf /var/lib/apt/lists/*
2326

24-
RUN add-apt-repository ppa:deadsnakes/ppa && \
25-
apt-get update && \
26-
apt-get install -y python3.9 python3.9-dev python3-pip && \
27-
rm -rf /var/lib/apt/lists/* && \
28-
python3 -m pip install pip --upgrade && \
29-
python3 -m pip install wheel
27+
RUN python3 -m pip install --no-cache-dir \
28+
--break-system-packages \
29+
awscli psycopg2-binary setuptools && \
30+
rm -rf /root/.cache /var/cache/apk/*
3031

3132
RUN apt-get update && apt-get install -y postgresql-client && \
3233
rm -rf /var/lib/apt/lists/*
3334

34-
RUN wget -c https://dl.google.com/go/go1.21.0.linux-amd64.tar.gz -O - | tar -xz -C /usr/local
35-
ENV PATH $PATH:/usr/local/go/bin
35+
RUN wget -c https://dl.google.com/go/go1.21.9.linux-amd64.tar.gz -O - | tar -xz -C /usr/local
36+
ENV PATH="${PATH}:/usr/local/go/bin"
3637

3738
ENV DATEFUNCTIONS_GITSHA=90d8d0f0daea4c8c5aa62edf440f26e9eb0ab950
3839
RUN git clone https://github.com/OpenHistoricalMap/DateFunctions-plpgsql.git /usr/local/datefunctions && cd /usr/local/datefunctions && git checkout $DATEFUNCTIONS_GITSHA
@@ -45,14 +46,11 @@ RUN git clone https://github.com/omniscale/imposm3.git $GOPATH/src/github.com/om
4546
WORKDIR $GOPATH/src/github.com/omniscale/imposm3
4647
RUN git checkout v0.14.2
4748

48-
4949
# Apply fixes or patches (if necessary)
5050
RUN sed -i '/setMaxFileSize/d' cache/ldb_pre_121.go
5151
RUN go install github.com/omniscale/imposm3/cmd/imposm
5252
ENV PATH $PATH:$GOPATH/bin
5353

54-
# Install AWS CLI for downloading files
55-
RUN pip3 install -U setuptools awscli psycopg2
5654
WORKDIR /osm
5755
COPY . .
5856
CMD ["./start.sh"]

‎images/tiler-imposm/config/imposm3.json‎

Lines changed: 23 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -339,12 +339,9 @@
339339
],
340340
"type": "relation_member",
341341
"filters": {
342-
"exclude_tags": [
343-
[
344-
"area",
345-
"yes"
346-
]
347-
]
342+
"reject": {
343+
"area": ["yes"]
344+
}
348345
},
349346
"mappings": {
350347
"railway": {
@@ -446,7 +443,7 @@
446443
"key": "end_date"
447444
},
448445
{
449-
"type": "pseudoarea",
446+
"type": "area",
450447
"name": "area",
451448
"key": null
452449
},
@@ -557,12 +554,9 @@
557554
]
558555
},
559556
"filters": {
560-
"exclude_tags": [
561-
[
562-
"boundary",
563-
"administrative"
564-
]
565-
]
557+
"reject": {
558+
"boundary": ["administrative"]
559+
}
566560
}
567561
},
568562
"water_areas": {
@@ -588,7 +582,7 @@
588582
"key": null
589583
},
590584
{
591-
"type": "pseudoarea",
585+
"type": "area",
592586
"name": "area",
593587
"key": null
594588
},
@@ -768,7 +762,7 @@
768762
"key": "end_date"
769763
},
770764
{
771-
"type": "pseudoarea",
765+
"type": "area",
772766
"name": "area",
773767
"key": null
774768
},
@@ -995,7 +989,7 @@
995989
"key": null
996990
},
997991
{
998-
"type": "pseudoarea",
992+
"type": "area",
999993
"name": "area",
1000994
"key": null
1001995
},
@@ -1172,7 +1166,7 @@
11721166
"key": "end_date"
11731167
},
11741168
{
1175-
"type": "pseudoarea",
1169+
"type": "area",
11761170
"name": "area",
11771171
"key": null
11781172
},
@@ -1534,12 +1528,9 @@
15341528
],
15351529
"type": "linestring",
15361530
"filters": {
1537-
"exclude_tags": [
1538-
[
1539-
"area",
1540-
"yes"
1541-
]
1542-
]
1531+
"reject": {
1532+
"area": ["yes"]
1533+
}
15431534
},
15441535
"mappings": {
15451536
"railway": {
@@ -1590,7 +1581,7 @@
15901581
"key": "name"
15911582
},
15921583
{
1593-
"type": "pseudoarea",
1584+
"type": "area",
15941585
"name": "area",
15951586
"key": null
15961587
},
@@ -1817,20 +1808,11 @@
18171808
],
18181809
"type": "linestring",
18191810
"filters": {
1820-
"exclude_tags": [
1821-
[
1822-
"area",
1823-
"yes"
1824-
],
1825-
[
1826-
"natural",
1827-
"coastline"
1828-
],
1829-
[
1830-
"boundary",
1831-
"administrative"
1832-
]
1833-
]
1811+
"reject": {
1812+
"area": ["yes"],
1813+
"natural": ["coastline"],
1814+
"boundary": ["administrative"]
1815+
}
18341816
},
18351817
"mappings": {
18361818
"landuse": {
@@ -1894,7 +1876,7 @@
18941876
"key": "end_date"
18951877
},
18961878
{
1897-
"type": "pseudoarea",
1879+
"type": "area",
18981880
"name": "area",
18991881
"key": null
19001882
},
@@ -1955,7 +1937,7 @@
19551937
"key": null
19561938
},
19571939
{
1958-
"type": "pseudoarea",
1940+
"type": "area",
19591941
"name": "area",
19601942
"key": null
19611943
},
@@ -2043,7 +2025,7 @@
20432025
"key": "end_date"
20442026
},
20452027
{
2046-
"type": "pseudoarea",
2028+
"type": "area",
20472029
"name": "area",
20482030
"key": null
20492031
},

‎images/tiler-imposm/config/layers/admin_areas.json‎

Lines changed: 2 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -7,47 +7,7 @@
77
"source:datetime"
88
]
99
},
10-
"generalized_tables": {
11-
"admin_areas_z0_2": {
12-
"source": "admin_areas",
13-
"tolerance": 5000,
14-
"sql_filter": "admin_level IN (1,2)"
15-
},
16-
17-
"admin_areas_z3_5": {
18-
"source": "admin_areas",
19-
"tolerance": 1000,
20-
"sql_filter": "admin_level IN (1,2,3,4)"
21-
},
22-
23-
"admin_areas_z6_7": {
24-
"source": "admin_areas",
25-
"tolerance": 200,
26-
"sql_filter": "admin_level IN (1,2,3,4,5,6)"
27-
},
28-
29-
"admin_areas_z8_9": {
30-
"source": "admin_areas",
31-
"tolerance": 100,
32-
"sql_filter": "admin_level IN (1,2,3,4,5,6,7,8,9)"
33-
},
34-
35-
"admin_areas_z10_12": {
36-
"source": "admin_areas",
37-
"tolerance": 20,
38-
"sql_filter": "admin_level IN (1,2,3,4,5,6,7,8,9,10)"
39-
},
40-
41-
"admin_areas_z13_15": {
42-
"source": "admin_areas",
43-
"tolerance": 5,
44-
"sql_filter": "admin_level IN (1,2,3,4,5,6,7,8,9,10)"
45-
},
46-
"admin_areas_z16_20": {
47-
"source": "admin_areas",
48-
"sql_filter": "admin_level IN (1,2,3,4,5,6,7,8,9,10)"
49-
}
50-
},
10+
"generalized_tables": {},
5111
"tables": {
5212
"admin_areas": {
5313
"fields": [
@@ -92,7 +52,7 @@
9252
"key": "end_date"
9353
},
9454
{
95-
"type": "pseudoarea",
55+
"type": "area",
9656
"name": "area",
9757
"key": null
9858
},

0 commit comments

Comments
 (0)