ArgumentError: invalid byte sequence in UTF-8

[stephenlawrence]

When I try to parse a document with non UTF-8 characters I get this:

irb(main):283:0> content = Plaintext::Resolver.new(file, document_file.upload_content_type).text
ArgumentError: invalid byte sequence in UTF-8
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/activesupport-4.1.7/lib/active_support/multibyte/chars.rb:172:in `codepoints'
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/activesupport-4.1.7/lib/active_support/multibyte/chars.rb:172:in `compose'
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/plaintext-0.1.0/lib/plaintext/resolver.rb:37:in `text'
	from (irb):283
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/console.rb:90:in `start'
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/console.rb:9:in `start'
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/commands_tasks.rb:69:in `console'
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
	from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands.rb:17:in `<top (required)>'
	from bin/rails:8:in `require'
	from bin/rails:8:in `<main>

[jkraemer]

what kind of document is that? Plaintext or CSV? In that case it's quite clear because the PlaintextHandler assumes UTF-8 as input encoding. We might either have to introduce a way to specify the encoding, leaving it up to the user to find out what that actually is for a given file, or we need some way to make an educated guess.

[stephenlawrence]

@jkraemer In this particular case its a MS Word document. Also happens with PDF.

[stephenlawrence]

Got this with a tiff file as well:

content = Plaintext::Resolver.new(file, df.upload_content_type).
Tesseract Open Source OCR Engine v3.03 with Leptonica
Cannot open input file: -dutf-8

[jkraemer]

would you mind providing sample files to reproduce these errors (the utf8 error as well as the tesseract error)?