Classification feedback

[willemarcel]

Hi @jremillard !! Congratulations again for this great work. I'll use this ticket to post some feedback.

Imports

Changesets didn't create anything and were classified as Import:
https://osmcha.mapbox.com/changesets/44070150
https://osmcha.mapbox.com/changesets/44067113

https://osmcha.mapbox.com/changesets/44054299 - it has a considerable number of deletions and modifications, so it shouldn't be considered as an import.

I use a logic to detect imports, mass modifications and mass deletions in OSMCha. Take a look on:
https://github.com/willemarcel/osmcha/blob/master/osmcha/changeset.py#L377

I consider more suspect the changesets that don't have a balance between the number of elements created, modified and deleted.

Spam

Some changesets with long comments and description tags are classified as spam, but are good edits:
https://osmcha.mapbox.com/changesets/43890576
https://osmcha.mapbox.com/changesets/44725218
https://osmcha.mapbox.com/changesets/43711394
https://osmcha.mapbox.com/changesets/44758435
https://osmcha.mapbox.com/changesets/43953212

Tagging error

This classification seems to work very well. Are you able to identify which features are tagged wrongly? It can be a bit difficult to the users to discover which feature has inappropriate tags in a changeset with a big number of features. In OSMCha we can add suspicion reason also to features (that way we can list the flagged features of the changeset).

Maybe you need to consider the name:xx variants, for example this changeset has not wrong tags, although it has a name tag with a value that is not 100% correct: https://osmcha.mapbox.com/changesets/44026170

Another point that I find useful is to flag a edit only if it adds a wrong tag. For example: https://osmcha.mapbox.com/changesets/44386947 modifies some features with tagging error, but the wrong tags were added in another edit. It would be great also to take in consideration the number of wrongly tagged features in a changeset.

Some tips

We have more problems with new users changesets, so it could be more useful to use the classification only to new users edits. For example, you could use it only with edits of users with less than 500 edits and less than 3 months of experience. We try to reduce the number of false positives and our great challenge is to save the time of the reviewers by filtering suspect changesets in the best way possible.

You can get the changesets reviewed as bad in OSMCha to train the classifier: https://osmcha.mapbox.com/api-docs/#!/changesets/changesets_harmful_list

[jremillard]

Hi @willemarcel , thank you for looking at this.

Imports

I agree with you that the import changesets you listed are not imports. I have not pushed it yet, but the current version gets those changesets correct.

In addition to looking at the ratio of add/modifies/delete, imports can also be detected by looking for tags that come from shape files, a geometric area tag, tags in all caps, and looking at the diversity of the tags added/modified versus the changeset count. Also, sometimes the changesets will say "import" in the description. Even a small changeset can be an import, and I am trying to flag them all.

SPAM

The SPAM detector is really tricky because SPAM is subjective.

43890576, 44725218 are SPAMy. The descriptions contain non-verifiable statements.
43711394,44758435, are not SPAM, the current version of the detector gets them correct.

I don't have many samples of SPAM. Honestly, I now think that SPAM in OSM is in practice incredibly rare. I will use the osmcha API to get your SPAM and imports changesets.

Tagging Error

changeset 44026170 does not have a tagging error. The software does correctly understand what tags were added in a specific changeset. So 44386947 is correctly marked as normal at least in the current version that is not pushed yet.

Changeset Diffs

If you look at the XML files in trainingdata\cache, I am storing real changeset diffs. The downside of doing this is that you can't classify a changeset without a history database to diff against, it is slow. It is much slower to reconstruct the changeset diff than run the classifier.

Tips

It might be possible to find what tag is causing the changeset to be marked as a tagging error by looking at the intensity of the activations in the ann. It would be some work to do this!

I agree about the accuracy and the workload caused by false detections. This classifier is getting about 4% of them wrong right now. With so many changesets a day, 4% wrong is much too high, it needs to be more like 1%. It is not good enough right now. I need more data, the model is quickly over fitting (only 4 epochs). Hopefully with the osmcha API, I can get more changesets to train on.

[willemarcel]

Nice, @jremillard ! When you have new results available, could you put the csv file anywhere and notify me? Thank you!