remove `removeAllZeroNumericPrefix` filter?

[missinglink]

Today I was investigating an issue where leading zeros in postcodes don't sort as expected.

After investigating some approaches to add additional should conditions, it occurred to me that maybe we can just get rid of this removeAllZeroNumericPrefix filter completely?

I honestly don't recall why we have that in the first place, it dates back over ten years to a huge refactor I did to move from an FST based index to an Edge N-Gram Index, the PR doesn't mention this functionality specifically.

@orangejulius also commented at some point he doesn't fully recall either.

Given the issues it causes to some postal codes and potentially venue names, would it be preferable to remove it as there are relatively few cases of leading zero housenumbers?

I'm possibly missing something, does anyone remember why, or can come up with some good reasons why we have that?

schema/settings.js

Lines 234 to 238 in 4d6d74d

	"removeAllZeroNumericPrefix" :{
	"type" : "pattern_replace",
	"pattern" : "^(0*)",
	"replacement" : ""
	},

Related:

• autocomplete does not return postal codes lower than 10000 pelias#898
• Add filter to remove leading zeros to peliasPhrase analyzer #475

cc/ @orangejulius @Joxit

[missinglink]

If we find that leading-zero housenumbers are in fact common in data and/or queries we may be able to handle that in a pre-processing function in both the importers and after the parser runs at query-time.

[missinglink]

Related test case (marked as failing):
https://github.com/pelias/acceptance-tests/blob/637a5bb18ae239c2026e3c5e4d442e7e520f5e12/test_cases/autocomplete_postalcodes.json#L91-L110

[missinglink]

/code/p/acceptance-tests master = !1 ❯ git grep '"0'
package.json:  "version": "0.0.0",
test_cases/address_matching.json:            "postalcode": "06410",
test_cases/address_parsing.json:            "postalcode": "06902",
test_cases/autocomplete_postalcodes.json:        "text": "03100",
test_cases/autocomplete_postalcodes.json:            "name": "03100",
test_cases/autocomplete_postalcodes.json:            "postalcode": "03100",
test_cases/autocomplete_postalcodes.json:        "text": "04106",
test_cases/autocomplete_postalcodes.json:            "name": "04106",
test_cases/autocomplete_postalcodes.json:            "postalcode": "04106",
test_cases/search_postalcodes.json:        "text": "03100",
test_cases/search_postalcodes.json:            "name": "03100",
test_cases/search_postalcodes.json:            "postalcode": "03100",
test_cases/search_postalcodes.json:        "text": "04106",
test_cases/search_postalcodes.json:            "name": "04106",
test_cases/search_postalcodes.json:            "postalcode": "04106",

[missinglink]

Worth noting that /v1/search appears to be working correctly:

Although this is actually just matching on the token "6000"

[orangejulius]

I think this behavior was written about in #475.

In that PR, we actually added more filtering to remove leading zeros, because we were inconsistent about it.

Looking at it now, I agree that removing leading zeros everwhere at the schema level is probably wrong. It's an attempt to account for one particular type of address data error: leading zeros in housenumbers.

I agree that a pre-processor that removes any invalid leading zeros from address housenumbers is something we should look at implementing. We should also investigate if there are valid cases of leading zeros in housenumbers. I hope not, but wouldn't be surprised and that would of course make things more complex.

If/when we decide we have leading zeros in housenumbers accounted for to a reasonable degree then we can look at merging this change?

[missinglink]

I suspect that the leading zero house numbers are more common in OpenAddresses where it comes from various sources rather than OpenStreetMap, which will likely also include a few.

In those cases it's an easy fix to either update the importers or pelias/model to strip the leading zero.

That way we can remove this general token filter and still maintain support for that feature if required.

[Joxit]

Nice catch, AFAIK in France we don't have leading zeros for horse numbers but as you showed in your example it's important for postal codes.

Maybe we could see on oa data all countries containing leading zero on house number and check if it's required? 🤔

[missinglink]

OK since no one remembers why we have it thinks we need to keep it, I'm going to remove it.

But first I will add a PR to pelias/model to post process leading zero housenumber so we don't regress in that area if it ever mattered.

[missinglink]

Okay, so I've updated the model (twice, since I found some bugs) and the {OA,OSM,CSV} importers.
I think #504 should be good to merge now.

[missinglink]

One thing worth noting is that since a housenumber such as 00005 will be indexed as 5 now, it will no longer be retrievable when entering 00005 at search time since the query-time analyzer will now leave it as-is.

This was previously possible, but I don't think it's an issue.