o1-preview bug for multibyte char

[adamist521]

Multibyte character is not shown right.
Works fine for gpt-4o-mini.

Changing the line to following would make rendering right for o1-preview... ( Nothing shown for gpt-4o-mini execution since there's no need to encode as iso-8859-1)

org-ai/org-ai-openai.el

Line 506 in 5adfde1

(insert (decode-coding-string text 'utf-8))

(insert (decode-coding-string (encode-coding-string text 'iso-8859-1) 'utf-8))

I couldn't figure out where text was encoded as iso-8859-1...

( Could be my emacs setup problem? Checked that coding-sytem was utf-8 by describe-current-coding-system
Are there any fixes you can think of?

Little detail on where encoding is going wrong

Checked the data returned from server in org-ai-stream-request was utf-8 but it looks like the data is encoded before org-ai--insert-stream-response

• org-ai-stream-request

  • org-ai/org-ai-openai.el

    Line 576 in 5adfde1

    (org-ai--url-request-on-change-function nil nil nil))

• org-ai--insert-stream-response

  • org-ai/org-ai-openai.el

    Line 533 in 5adfde1

    (cl-defun org-ai-stream-request (&optional &key prompt messages model max-tokens temperature top-p frequency-penalty presence-penalty service callback)

sample screenshot

Input: 日本語いけますか?

Before patching the line.

o1-preview

gpt-4o-mini

After patching the line

o1-preview

gpt-4o-mini

Could be related

#132

[torgeir]

Being norwegian, I experience the same behavior.

Also, the suggested replacement (insert (decode-coding-string (encode-coding-string text 'iso-8859-1) 'utf-8)) works for o3-mini, but does not work for gpt-4o-mini.