expand_address hangs with certain strings, with invalid UTF-8 warnings

[nitinvijay94]

Hi

Firstly, thanks for all the work you have done. In order to avoid fluff, I'll be posting the context serially.

1. United States
2. I'm using Libpostal to dedupe addresses within the Hadoop ecosystem (Hive on Tez).
3. I have a farily large set of over 200 million addresses, a size-able chunk of which are human entered values. Given the nature of my data, I have encountered a few cases which causes the expand_address function to hang and stop my job.

a) The most baffling case.

>>> expand_address(u'5-19�� Nakamachi')
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None

this is an entirely ASCII string, which halts the program. Using parse_address also throws warnings, but continues gracefully.

>>> parse_address(u'5-19�� Nakamachi')
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
WARN  invalid UTF-8
   at transliterate (transliterate.c:791) errno: None
[(u'5-19&#56256 &#56321 Nakamachi', u'house')]

b) Address: "No. \uD835\uDFE3\uD835\uDFE3"
This looks like "No. 11". Works fine using pypostal, however, it similarly halts the program when using jpostal. My guess is this has something to do with the C interface's GetStringUTFChars not working well with 4 byte utf-8 characters, since Java converts its internal UTF-16 String to a Modified UTF-8 format.

These cases are rare, but can block processes, which makes them problematic. Is there some way we can have this function exit gracefully in case of utf-8 parsing errors?

Thanks,
Nitin

[james-willis]

I ran into this issue today doing something very similar to what you describe here. The issue is occurring specifically as part of the numex expantion.

In your first case the HTML escaped characters get converted in normalize.c:226

DEBUG original_str=5-19�� Nakamachi
DEBUG new_str = 5-19�� Nakamachi

Somewhere in convert_numeric_expressions the thread hangs. I will continue to debug this but I think probably it would be best to drop these characters from the string rather than make the normalize/numex code robust to them

Your second test case winds up with the same replacement character:

DEBUG  transliteration complete, str=No. ����, len=16

[james-willis]

unicode_scripts.c:46 calls utf8proc_iterate_reversed to get the value of char_len which calls utf8proc_iterate. utf8proc_iterate returns a negative number when a character is not valid utf8 which may cause an infinite loop in some cases.

[albarrentine]

Fixed this in #703. Reviewing all the calls to utf8proc_iterate, there were a few that should be checking return codes and aborting mission if invalid UTF-8 is detected.

Part of the contract with the C library is that the user (typically the binding unless using C directly) is responsible for making sure the input is valid UTF-8 before sending through to the C functions. Java's the only place I know of that has issues there because of Java's aforementioned Modified UTF-8 format. There is a PR on jpostal openvenues/jpostal#38 and a slightly modified one on OvertureMaps' fork OvertureMaps/jpostal#1 to convert it to real UTF-8, the only issue being at one time they were breaking older JDK versions. Now that the requirement is JDK 16+, I'm happy to merge one of those if anyone's interested in updating.

For Python, there's an excellent package called ftfy (fixes text for you), which fixes Mojibake, weird encodings, and other invalid text before passing text to libpostal. It's a simple function call and will fix both of the above cases and probably many others. It won't be as fast as libpostal to run over 200M addresses but will ensure correct input (can also just sanitize the data once on the way into the cluster, that probably helps with a number of encoding issues that will happen in Python or any other language processing truly raw input bytes).

The first test appears to be two invalid unicode codepoints U+56256 and U+ 56321 that should just be converted to the unicode replacement character '\uFFFD' by ftfy or other sanitization libraries if they handle HTML entities, but since libpostal also will convert any HTML entities it finds internally, we also have to handle the case where invalid UTF-8 can be injected into a string via said entities.

The second test is encoded using Unicode surrogate pairs which ftfy will pair correctly if possible.

>>> print('No. \uD835\uDFE3\uD835\uDFE3') # Notice we can't even print this string in Python
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 4-7: surrogates not allowed
>>> ftfy.ftfy('No. \uD835\uDFE3\uD835\uDFE3')
'No. 𝟣𝟣'
>>> ftfy.fix_and_explain('No. \uD835\uDFE3\uD835\uDFE3')
ExplainedText(text='No. 𝟣𝟣', explanation=[('apply', 'fix_surrogates')])

I suppose we should probably have something in libpostal to fix surrogates which can still be encoded by the corresponding HTML entities, but with the changes in #703 those will at least be identified as invalid UTF-8 and won't cause any unterminated loops. I'll ask the Java world to pick which PR to merge to handle correctness via converting Modified UTF-8 to standard UTF-8.