We should strive to give a more informative message regarding incorrect input encoding. Minimally, we should point the user to the fileand recode utilities. We could also consider allowing them to specify the input encoding as in ./parser_wrapper.sh -e latin1 and pass that to the input sanitizer script.
I would not attempt to autodetect the encoding, though. The libraries needed for that don't seem to be available by default:
http://stackoverflow.com/questions/436220/python-is-there-a-way-to-determine-the-encoding-of-text-file
@jmnybl @fginter
We should strive to give a more informative message regarding incorrect input encoding. Minimally, we should point the user to the
fileandrecodeutilities. We could also consider allowing them to specify the input encoding as in./parser_wrapper.sh -e latin1and pass that to the input sanitizer script.I would not attempt to autodetect the encoding, though. The libraries needed for that don't seem to be available by default:
http://stackoverflow.com/questions/436220/python-is-there-a-way-to-determine-the-encoding-of-text-file
@jmnybl @fginter