fzf on Windows assumes UTF8 stdin, but it should assume CP_ACP stdin (current system codepage)

[chrisant996]

Checklist

• I have read through the manual page (man fzf)
• I have searched through the existing issues
• For bug reports, I have checked if the bug is reproducible in the latest version of fzf

Output of fzf --version

0.55.0 (fc69308)

OS

• Linux
• macOS
• Windows
• Etc.

Shell

• bash
• zsh
• fish

Problem / Steps to reproduce

1. Use any system codepage other than 65001 (which is Beta and is never the default).
2. Run dir | fzf when any file names are present with non-ASCII characters.

Result = fzf interprets the stdin as UTF8, but the stdin is never UTF8 by default, and so fzf shows garbled text.
Expected = fzf interprets the stdin as being in the system codepage by default, like native Windows programs do.

Something in fzf is going out of its way to assume UTF8 stdin on Windows, but that isn't how Windows programs work.

Ideally fzf would assume system codepage by default, and could have something like a --utf8 flag to select assuming UTF8 stdin.

[junegunn]

Thanks for the pointers.

Here's the background.

1. fzf was originally written for *nix systems only using ncurses.
2. Windows support was added by @kelleyma49 using the tcell library. We still use it on Windows in full-screen mode (https://github.com/junegunn/fzf/blob/master/src/tui/tcell.go), and I believe this implementation properly handles I/O using native Windows system calls.
3. fzf added --height option to enable non-full-screen mode. We added "LightRenderer" which reads from and writes to /dev/tty directly. No Windows support for --height at the moment.
4. @kelleyma49 sent a patch to enable --height option on Windows. From my understanding, it changed the LightRenderer to read from CONIN$ and write to CONOUT$ instead of /dev/tty on Windows. But it's a thin adapter, and as you mentioned, it assumes that the same byte sequences as *nix systems are used.
5. I wanted to get rid of tcell-based renderer and use LightRenderer even in full-screen mode to simplify the code and for better mouse support on mintty (dca2262)
6. Then we realized that LightRenderer cannot properly read non-ASCII characters, and had to revert the decision.
7. @kelleyma49 confirmed when the code page is set to 65001, fzf using LightRenderer (--height) can properly read non-ASCII characters on Windows Terminal (Set console to UTF-8 encoding on Windows #3951). The patch to set the code page to 65001 has not been merged yet. I'm not quite sure if it's the right way to fix the problem.

Because I'm not a Windows user and have a limited understanding of the system, I don't think I'll be able to fix the problem myself. I'm hoping that someone will contribute.

[chrisant996]

it changed the LightRenderer to read from CONIN$ and write to CONOUT$ instead of /dev/tty on Windows. But it's a thin adapter, and as you mentioned, it assumes that the same byte sequences as *nix systems are used.

Yup, that'll only work when the console code page is 65001 (UTF8).

But I don't know whether that can handle function keys, arrow keys, and other extended keys, etc. I've never tried doing file IO reads from CONIN$ when reading keyboard input, since Windows hasn't had VT input emulation until very recently (partway thru the Win10 lifecycle), and programs typically want to be backward compatible with older versions of Windows.

The normal approach for reading keyboard input is to use ReadConsoleInputW on CONIN$, which returns a struct with details about a keyboard/mouse/resize/etc event (depending on console mode flags on the conin handle). On Win11 with the right console mode flag set on the console input handle, then I expect using ReadFile (like the patch currently does) would work. If you want to support older versions of Windows (which in my opinion is probably NOT necessary for fzf), then it would be necessary to use ReadConsoleInputW instead -- but I think that's probably not worth implementing.

7. @kelleyma49 confirmed when the code page is set to 65001, fzf using LightRenderer (--height) can properly read non-ASCII characters on Windows Terminal (Set console to UTF-8 encoding on Windows #3951). The patch to set the code page to 65001 has not been merged yet. I'm not quite sure if it's the right way to fix the problem.

As long as it's careful to intercept Ctrl-Break and Ctrl-C and restore the original code page, then that should be ok.

Or, there's a chance that maybe each console input handle has separate mode bits (SetConsoleMode). If so, then it might not even be necessary to restore the console mode. I'll run some tests to find out, and reply here with the results.

UPDATE: Yes, any console modes and the console code page must be restored before fzf exits. If fzf isn't already intercepting Ctrl-Break/Ctrl-C and restoring the mode(s) and CP, then let me know and I can share how to hook that up.

Because I'm not a Windows user and have a limited understanding of the system, I don't think I'll be able to fix the problem myself. I'm hoping that someone will contribute.

Understood. I don't know Go at all, but I have decades of deep knowledge of Win32 APIs and system design, and C/C++ experience. So at the very least, I can offer "consultation services".

There are multiple ways to support Unicode keyboard input, so I think that can get solved soon, and probably using SetConsoleCP is sufficient, as long as the original CP is restored before fzf exits.

But I'm even more interested in the stdin encoding issue. It's unrelated to keyboard input, and addressing it should just be a matter of parsing line breaks, and then for each line call MultiByteToWideChar once from CP_ACP, then call WideCharToMultiByte once using CP_UTF8.

I'm willing to try to learn Go enough to at least write some pseudocode, or maybe even get it compiling and testable. Can someone share a code pointer to where stdin is read?

Windows console (text mode) programs write 8-bit output encoded in ACP (GetACP()). It's problematic to require callers to externally pipe stdout into some conversation tool, and pipe that into fzf stdin -- partly because Windows doesn't have such a tool (there's no built-in analog to iconv, though various third party tools exist). It would be very beneficial if fzf could (on Windows) default to expecting stdin encoded as ACP, and internally convert it to UTF8. And since some programs have a special flag such as --utf8 to force output as UTF8, it would be great for fzf to have a flag --utf8 that reverts to the current behavior where it assumes all input is UTF8. Ideally fzf would do the same conversion in reverse on its stdout as well, also subject to the --utf8 flag.

[chrisant996]

@junegunn just to be clear -- I opened this issue about stdin encoding, not about keyboard input encoding. They're separate things on Windows.

[chrisant996]

Also, a caveat ... I said stdin/stdout use GetACP() encoding on Windows.

That's an oversimplification and is a bit inaccurate.

In reality console mode apps should emit stdout using the GetConsoleOutputCP() encoding.
But since GUI apps must use GetACP() instead of GetConsoleOutputCP(), some console mode apps mistakenly use GetACP().

The OEM codepages are very similar to normal codepages, but have some different legacy characters. E.g. 1252 is the English codepage but 437 is the English OEM codepage (I'm oversimplifying, but GetConsoleOutputCP() reports the actual codepage, whatever it is).

GetConsoleOutputCP() should be the right general default, especially since cmd.exe's dir command uses GetConsoleOutputCP(), not GetACP(). And the dir command is the obvious choice when collecting filenames for completion on Windows (since dir is built into the OS cmd.exe command shell, and doesn't require any additional programs to be installed).

So...

• The default should be to use GetConsoleOutputCP() as the encoding for both reading from stdin, and writing to stdout.
• Ideally some --utf8 flag would override the default and use CP_UTF8 instead.
• It might also be useful to have a way to manually override the codepage (esp. for using GetACP() and/or CP_ACP in the MultiByteToWideChar and WideCharToMultiByte functions).

Does that sound like a reasonable design?

[skarasov]

Win 10, pwsh 7.4.6, fzf 0.56.0

As workaround use PowerShell 😊
First pic - beta utf8 support turned off.

Second pic - turned on.

[chrisant996]

Win 10, pwsh 7.4.6, fzf 0.56.0

As workaround use PowerShell

Powershell does not by default use UTF8 as the IO encoding.

You must have configured your Powershell to set $OutputEncoding to UTF8.

Yes, there are workarounds. But it's reasonable for me to help update fzf to support encodings the same way native Windows programs do. Then things will just work, without needing to find/use workarounds (which may be different for different programs).

Default encoding in Powershell is not UTF8:

Yes it's possible to force Powershell to use UTF8 (or some other encoding).

[skarasov]

In PowerShell 7.4.6 (not Windows PowerShell 😒) $OutputEncoding defaults to utf-8.

And, imho, stop feeding Microsoft legacy.

[chrisant996]

In PowerShell 7.4.6 (not Windows PowerShell 😒) $OutputEncoding defaults to utf-8.

I thought Powershell 7.4.6 is not installed by default, right?
So doesn't the suggested workaround require installing an additional program?

And, imho, stop feeding Microsoft legacy.

That doesn't feel very empathetic to people who use Windows.
I'm trying to support users and get things working automatically.
It doesn't come off well to suggest that they should just quit using Microsoft products, or do a bunch of workarounds (which are different for each program that pipes output into fzf).

Why can't we make it Just Work?

[skarasov]

I thought Powershell 7.4.6 is not installed by default, right?

fzf is not installed by default, right?

very empathetic to people ... support users ... doesn't come off well

Not a word about users from my side. Don't misinterpret. Intention - stop feeding legacy - sooner UTF-8 will be default in Win.

Why can't we make it Just Work?

Because of this we more than 15 years struggling with different codepages rather one.

And just chcp 65001 before dir | fzf.

[chrisant996]

I thought Powershell 7.4.6 is not installed by default, right?

fzf is not installed by default, right?

I have users of Clink who install fzf. And then they're surprised that Fzf doesn't work with non-ASCII content by default, they have to either install extra programs or do a bunch of research and workarounds to be able to use fzf.

Out of the box, fzf doesn't play very well with Windows. I'm offering to junegunn to help make fzf work better out of the box on Windows. Your suggestions are beside the point of the issue. I appreciate them, and may use some of them, but I'd like to keep the discussion focused on the stated issue.

very empathetic to people ... support users ... doesn't come off well

Not a word about users from my side. Don't misinterpret. Intention - stop feeding legacy - sooner UTF-8 will be default in Win.

I understand your viewpoint.

Why can't we make it Just Work?

Because of this we more than 15 years struggling with different codepages rather one.

And just chcp 65001 before dir | fzf.

I understand that you disagree with making any change to fzf, and that changes should be external to fzf for Windows, to try to change how Windows works.

That doesn't mean junegunn and I cannot make fzf work seamlessly on Windows. It only means you prefer not to.

[chrisant996]

If junegunn is not willing, then I'll drop the topic. But I'd like to have the conversation about what it would take to do it. I think it's a pretty easy change to make.

I'll grant that maybe the standalone Powershell using UTF8 by default means fzf should continue to default to expecting UTF8. But adding a flag to accept the system code page (or even arbitrary encodings) is not unreasonable. There are plenty of Linux programs that explicitly have flags to control both input and output encodings. The concept is very compatible with Linux.