Add moderation zones where anonymous notes are not allowed

[pablobm]

Fixes #6570 (Enable PostGIS).

This PR introduces "moderation zones" where anonymous notes are not allowed. It also enables PostGIS in the process.

This addresses #6567 partially:

• Within the moderation zones, only unauthenticated users are blocked, not "recent" ones. We can measure the success of this one before moving on to recent accounts which are a bit more complex.
• No UI for moderators to define the zones. This will be implemented in a future PR.

Interacting with PostGIS

In PostGIS terms, these zones are defined as geometries as opposed to geographies. This is consistent with the rest of the API.

Two gem options are available:

• activerecord-postgis-adapter:
  • "Classic" gem, 15 years old.
  • Requires setting adapter: postgis which can be a headache when moving between branches.
  • Doesn't work well when using geography-enabled functions such as ST_Covers. Probably can be added by implementing the operations with Arel.
  • Does work well with ST_Contains.
  • Works with Ruby 3.2, probably lower but I haven't checked.
• activerecord-postgis (current choice):
  • Very recent gem, 1 year old, I only discovered it by accident.
  • Doesn't require changing config/database.yml.
  • Works with geography-enabled functions such as ST_Covers by interpolating params into ?, but not with Arel (can be added though).
  • Requires Ruby 3.3 minimum (although as mentioned Ruby 3.2 is about to enter EOL).
  • Claims not to be based on monkeypatching in comparison to the competition (I haven't looked into this).

In common to both:

• They are drop-in replacements for each other, at least as long as you stick to geometry functions.
• Both support Arel queries for geometry, which is the way I have implemented the query.
• They use RGeo behind the scenes to encapsulate GIS objects. This requires creating them with one of several available factories. I have gone for RGeo::Cartesian.simple_factory as it appears to just work. If we changed to one of the others, we'd need the gem ffi-geos. This is not made clear in the documentation.

I haven't found a big argument for either, but I have gone for the newer one if only because of the simplicity of not having to update config/database.yml.

Future work

• If an unauthenticated user writes a long, thoughtful note, submits it, but then it fails because it was a moderation zone... they have wasted their time, if not lost their work. Therefore we may want a way for clients to tell ahead of time if a specific location has a block. Probably an API such as GET /api/0.6/notes/allowed?lat=X&lon=Y.
• There's no UI to define the zones. We'll need a CRUD for moderators to use.

[mmd-osm]

Geoblocking usually refers to a different concept, i.e. a block based on the geographical location of a user, rather than an area they aren't supposed to edit in. I believe it would avoid confusion if this feature were given a different name.

[pablobm]

it would avoid confusion if this feature were given a different name.

Good call. Happy to hear suggestions 🙂

[gravitystorm]

Initial thoughts inline.

On a general overview, I'm happy with the scope of this initial PR, and then following up with a separate PR to sort out the "detection api" topic, then actually creating the zones via a UI.

[gravitystorm]

You asked about which gem to use, but I don't have a strong opinion on this, other people might have more experience with the alternative gem. It would be good to check if they are easily interchangeable - i.e. is the reason for their difference more to do with monkeypatching, or do they also have e.g. differences in how to write the queries?

[pablobm]

This is what I have so far:

1. activerecord-postgis-adapter: this seems to be the popular choice, but I'm having issues with it and barely can make it work. Also I find it annoying that I need to change the DB config to adapter: postgis, as then I have to change it back when I move branches to look at something else.
2. activerecord-postgis: it works and doesn't have the adapter: annoyance. However it requires Ruby 3.3 minimum.

For now I have gone with option 2. I need to research what's wrong with option 1 and whether I'm doing something wrong.

The differences appear to be with the implementation, but I haven't looked much into it yet. They seem to be interchangeable, with queries written pretty much the seem way and both relying on rgeo-activerecord.

[tomhughes]

Ruby 3.2 is EOL at the end of March in any case so we probably need to be thinking about moving to 3.2 as a baseline.

[pablobm]

Trying to debug this. If I use activerecord-postgis-adapter, with this very same code, I get this warning and error:

$ bin/rails test test/models/geoblock_zone_test.rb
Running 1 tests in a single process (parallelization threshold is 50)
Run options: --seed 64084

# Running:

unknown OID 147959: failed to recognize type of 'zone'. It will be treated as String.
E

Error:
GeoblockZoneTest#test_falls_within_any:
TypeError: Cannot visit RGeo::Cartesian::PointImpl
    app/models/geoblock_zone.rb:38:in 'GeoblockZone.falls_within_any?'
    test/models/geoblock_zone_test.rb:10:in 'GeoblockZoneTest#test_falls_within_any'

However if I don't have this problem in a new Rails app, with same gem versions, model, migration, and test.

So there's something in our gems or config that is interfering with activerecord-postgis-adapter but has no issues with activerecord-postgis.

[pablobm]

Trying to bisect this a bit. It's tricky. For now I know that revision aad81eb shows the same behaviour. I can't compare it with the alternative gem because it requires Ruby 3.3.0, which I can't get working there.

[pablobm]

The last issue was probably me. I think I had forgotten at one point to set adapter: postgis. Summary:

• activerecord-postgis-adapter: "classic" gem, 15 years old. Requires setting adapter: postgis which as I just illustrated can be a headache. Doesn't work well when using geography-enabled functions such as ST_Covers. Does work well with ST_Contains. Works with Ruby 3.2, probably lower but I haven't checked.
• activerecord-postgis: very recent gem, 1 year old, only discovered it by accident. Doesn't require changing config/database.yml. Works with geography-enabled functions such as ST_Covers interpolating ?, but not with Arel (can be added though). Requires Ruby 3.3 minimum (although as mentioned Ruby 3.2 is about to enter EOL). Claims not to be based on monkeypatching in comparison to the competition (I haven't looked into this).

My preference is for the newer gem, if only because of the database.yml thing that does bother me, but I appreciate is only a short term thing. I'm sensing that dropping Ruby 3.2 would be on the cards for maintainers?

[gravitystorm]

We want to get geography vs geometry sorted out here, and this is something I'd like feedback from other people on.

I lean slightly towards geometry, for both theoretical and practical reasons. For example, let's say we draw a rectangular geoblock_zone, with four corners and a top corners at 70°N. For a geometry, the top edge is along the 70°N parallel. But for a geography, the edges of shapes are defined by great-circle arcs between the corners, and this might be unexpected behaviour. Halfway between the top corners, the arc could include 72°N, for example, or go over the pole if the corners are 180° longitude apart. Is this way moderators and mappers would expect?

We (generally?) treat OSM coordinates as a rectangular grid, and ways are (typically?) drawn as straight lines on a plane, rather than great-circle arcs.

So geography is a valid option, but I'm not sure if the extra complexity is worth it. One example of complexity is that many fewer PostGIS functions are implemented for geography compared to geometry.

[pnorman]

We want geometry to have behavior consistent with the rest of the API

[pablobm]

Any preference of factory to use? https://github.com/rgeo/rgeo/blob/main/doc/Which-factory-should-I-use.md I am using RGeo::Cartesian.simple_factory for the moment as it appears to "just work". If we changed to one of the others, we'd need the gem ffi-geos.

[pablobm]

Forgot to mention: I have changed this to be geometry.

[pablobm]

Any ideas on naming? If the current geoblock_zones is not good, options could be: zone_blocks, protected_zones. Also areas instead of zones. Thoughts?

I'm leaning towards protected_zones as "block" could be understood as a noun and cause confusion due to its other meanings.

[gravitystorm]

Also areas instead of zones

Probably not areas, since they have a meaning already in OSM. protected_area also has a specific meaning.

[mmd-osm]

On the proposed API 0.7 page, someone called them "lock regions". I think that's not too bad. The other proposal "Quarantine" sounds a bit too strong for what we want to achieve here.

https://wiki.openstreetmap.org/wiki/API_v0.7#New_Features

[pablobm]

"lock regions". I think that's not too bad

"Locked regions" (if I may change the grammar slightly) could work. Having said that: even if the feature proposed in the wiki may never exist, I'm conscious that we might be overloading the term. I'm thinking whether the concept could be used for both this and what's described in the wiki and I think yes it could, but I could be missing something.

[simonpoole]

May I suggest that doing this with postgis would only make sense if you are considering arbitrary boundaries. If you are just considering blocking at a country/state level there are a number of super fast geocoders (including one used by iD) that do this without all the complexity.

I would note that I'm very much for moving to postgis for other functionality, but for blocking notes it seems a tiny bit of overkill I might say.

[1ec5]

I think the discussions that led us here were focused on arbitrary regions, or at least proximity to a certain location. Blocking an administrative boundary per se might be of less interest for notes specifically, but I could see that being potentially relevant for other kinds of blocking.

super fast geocoders (including one used by iD)

To elaborate, iD uses country-coder, which stores simplified country boundaries in a which-polygon spatial index, which in turn is based on RBush, which is an R-tree implementation, like PostGIS.

[simonpoole]

To elaborate, iD uses country-coder, which stores simplified country boundaries in a which-polygon spatial index, which in turn is based on RBush, which is an R-tree implementation, like PostGIS.

I wasn't implying that the iD implementation was the best in class (it's the worst), just that it isn't a novel idea even for OSM central infrastructure.

[pablobm]

only make sense if you are considering arbitrary boundaries

It's for arbitrary boundaries indeed. For the sake of simplicity and separation of concerns, this PR doesn't include a tool to edit the boundaries, but the plan is to add one at a later stage.

Additionally, adding PostGIS support now would break a psychological barrier (if you will) that will encourage the creation of features that benefit from it.

[pablobm]

Another potential name: moderation zones.

[pablobm]

There's at least the question of naming open, but I think this advanced enough now to be out of draft status.

[pablobm]

I have renamed the model to ModerationZone and tweaked a couple of things.

[pablobm]

I have rebased after merging the PR above. This works well now.

• I have added a file .devcontainer/Dockerfile.postgres, similar to the existing docker/postgres/Dockerfile, which installs Postgis using Apt. This works around Postgis not having images for arm64 (at least not official ones).
• Both containerised setups work on my arm64 and amd64 machines.

[gravitystorm]

Great. I think it's just the ruby min version thing that's blocking this now for me. Ways forward are discussed above.

[tomhughes]

The fact that it will not be deployable in production is not a blocker?

[gravitystorm]

The fact that it will not be deployable in production is not a blocker?

Do you mean that nobody has added package "postgresql-15-postgis-3" to the OSMF chef repo, or are you warning us that there's some more deep-seated reason that this cannot be deployed?

[tomhughes]

I mean, as previously explained, that a migration that tries to enable postgis needs to run as a superuser and we do not run migrations as a superuser currently and it's not clear that we easily can if we assume that we don't want the production rails code to be using a superuser account!

[gravitystorm]

a migration that tries to enable postgis needs to run as a superuser

I've tested locally that "CREATE EXTENSION IF NOT EXISTS" as a non-superuser will succeed, if the extension already exists. Could this be a way forward - i.e. to create the extension via chef as a superuser, and then run the migrations in this PR as normal?

[tomhughes]

Maybe, it's sort of what I did to get the test instance deployed on dev except there I also inserted the schema migration record manually.

The other option would be to create a separate user for running migrations but that would need a bit of work especially in the dev environment and would require some way to override the database connection details when running the migrations.

There's also the question of how it impacts developers and third party users of the code.

[pablobm]

Great. I think it's just the ruby min version thing that's blocking this now for me. Ways forward are discussed above.

Sounds good, I have now implemented my preferred of the ways forward I proposed above. Specifically: I have created a fork of the gem, allowing for Ruby 3.2. This is following the conversation with the author (see comment linked above and subsequents) where it seems unlikely that our app will be affected by the Ruby 3.2 incompatibility.

The fact that it will not be deployable in production is not a blocker?

Happy to discuss what can be done here. I did ask at #6713 (comment)

[tomhughes]

If I had a good solution I would have offered it when you asked - the problem is that I don't.

[woodpeck]

Note, DWG currently runs a script that auto-hides anonymous notes in certain regions, which would be superseded by an API functionality like the one described here. The current DWG implementation can only deal with rectangular areas for which the point-in-polygon test is trivial and does not require PostGIS. Not saying that you shouldn't use PostGIS but if it turns out to be a pain, rectangles are your friend.

[pnorman]

I know a comment of this nature is best done early into the review process, but I've been looking over this issue and #6570 and can't find where PostGIS was decided as the best way to do this. Are we satisfied that it's the best option?

[pablobm]

Still better asked now than after merge! 😄

My view is that we are introducing PostGIS here as a strategic move. This PR adds an ancillary feature that can serve as a first, localised use case for PostGIS without affecting OSM at large.

From conversations, I understand that over the years features have been proposed that either required PostGIS or have received a sub-par implementation due to the lack of it. Introducing PostGIS at last breaks a psychological barrier that will allow the creation of new and more performant features in the future.

[pablobm]

There's also the question of how it impacts developers and third party users of the code.

This is an interesting point. I have not had issues in any of the environments I've tried, but I understand there will be use cases that escape me.

One specific case I'm wondering about: @hlfan, I believe you work on Windows? Would you be able to test this PR and tell us if you had any issues getting PostGIS working?

@tomhughes - Are there any specific use cases that you have in mind?