Replace Unicode characters with ASCII equivalents on paste

[rowanseymour]

For example:

• “ (U210C) and ” (U201D) to " (X22)
• ‘ (U2018) and ’ (U2019) to ' (X27)
• … (U2026) to ... (X2E X2E X2E)

To reduce chances of message needing to be sent as Unicode

[rudigiesler]

Interested to see how this gets dealt with. There are some nuances, eg. for "rich" channels like FB messenger and WhatsApp, you probably want the unicode characters, but for SMS channels, you probably don't. (but there are also special characters in the GSM character set, that are not in ASCII, complicating the matter)

In case it helps, for our projects where we never want to send any unicode, we pass message content through https://pypi.org/project/Unidecode/

[rowanseymour]

Yeah definitely open to debate here. The thinking behind the original ticket was that users are copy and pasting stuff from something like Word and don't realise they're including characters which might cut the message length in half. So one option would be to do these replacements only on paste, and then let users re-add whatever characters they want.

Another option would be to just inform the user that the message contains non-GSM characters: #745 (comment)

[rowanseymour]

Another angle to this is that users sometimes end up with these quotes inside expressions and then the expression fails to evaluate.