Arabic, chinese, etc. place names searchable

[klokan]

In this moment it seems like the non-latin names are ignored - the search results are not delivered.

There are street names in OpenStreetNames available only in local language scripts - which you can't discover with OSMNames search, even if they are in the TSV export now (and are going to be in the "alternative_names" columns.

Example:

 "name": "Guelmim كلميم",

discoverable under "Guelmim" but not discoverable under "كلميم".
Surprisingly cyrilic characters are index it seems - as there are some results for "Москва".

[klokan]

We may need to improve the charset_tables
http://sphinxsearch.com/docs/current.html#conf-charset-table
Related:
http://sphinxsearch.com/wiki/doku.php?id=charset_tables

[klokan]

An alternative would be transliteration before indexing / and before search:
unidecode see https://pypi.python.org/pypi/Unidecode

Used at https://github.com/mapbox/carmen#handling-non-latin-text

[klokan]

unidecode before indexing is the way to go.