Skip to content

Integrate database - #50

Open
makkos-lilly wants to merge 5 commits into
mainfrom
integrate_database
Open

Integrate database#50
makkos-lilly wants to merge 5 commits into
mainfrom
integrate_database

Conversation

@makkos-lilly

Copy link
Copy Markdown
Collaborator

Completed integrating the database with the OSM dashboard.

  • Added a python overlay file (overlay.py) that builds a project-local overlay in data.db_overlay from the database. This overlay file filters to users in data/metadata/group_users.csv with optional extra filters. It writes changesets_subsets.csv, changesets_tags_subset.csv, contributions_summary_subset.csv, user_hashtag_pairs.csv and _overlay_log.txt to data/db_overlay. The file contains robust path resolution so the overlay can be called from the correct folder. 
  • Added a new data bootstrap to the dashboard file that detects the data root and then reads either the database overlay (from data/db_overlay when use_db_overlay:true) or the raw/API path (from data/raw when use_db_overlay:false). It also creates robust bounding box validation for the changesets. 
  • Hashtags and all user changeset information now come from the database. Changesets_tags_subset.csv is used to populate information about user hashtags. The word cloud is capped at the top 50 tokens. 
  • In the dashboard, three extra boxes are added for more fine-grained information on user comments and hashtags. Previously, we just had the comment bigrams. These are: top words, per-user TF-IDF view, raw comments capped at 20 most recent. 
  • The feature keys panel is only available when use_db_overlay:false as keys such as highway, building, etc are only available from the planet file, which is accessed via APIs. When use_db_overlay:true, these panels are not available.
  • README file is updated for information on how to use the dashboard with the database (using use_db_overlay:true when rendering the dashboard with quarto). 
  • To use the dashboard with the database the database must be named osm_changesets.duckdb and must be placed in the directory named database.

To run the dashboard using the database, when rendering the dashboard write "quarto render dashboard.qmd -P use_db_overlay: true"

Creating a python environment from which to run the database connection and database queries. Attempting to create optionals to populate some .csv files with changeset details from the database. This does not work yet as details from the database are not compatible with some requirements.
Execution for dashboard build is continuously halting. All functions from dashboard.qmd have been moved to database.R. Additional files have been added to .gitignore and 'Rbuildignore to avoid unnecessary fatal build errors.
Fixed errors of differing fields between database vs API retrieval. Added db_overlay text to monitor correct retrieval from database. Added three boxes for more fine-grained information on user comments and hashtags. Checked database outputs against online changeset data. Ensured everything not filled in by the database is filled in by the previous API calls (apart from keys).
@makkos-lilly makkos-lilly linked an issue Sep 27, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

consider retrieving changesets info from OSM-hasthag-explorer

2 participants