Unicode support

[fmang]

The task elevated_shell.ps1 creates runs cmd.exe, which calls PowerShell with its outputs redirected. By having cmd.exe make the redirections, the files are encoded as OEM.

If we dropped the cmd step and ran PowerShell directly, the whole data pipeline would have full Unicode support.

Performing redirections in PowerShell with Write-Host involved is a nightmare, especially if we want to support older PowerShell versions like 4. I haven’t had any luck, but if someone manages to have the task spawn PowerShell directly without breaking the test suite, glory to them!

Related to #43 which fixes encoding corruptions, but limits the supported character set to OEM/ANSI.

[mwrock]

Its been so long that I really can't remember why we did not go directly through powershell and went to cmd first. I'm pretty sure we tried and ran into issues similar to yourself.

[fmang]

I see. The trick with the cmd wrapper is that it merges the multitude of PowerShell streams into stdout/stderr. Since PowerShell wouldn’t collaborate, I thought I could use System.Diagnostics.Process directly to emulate cmd’s behavior, and I managed to get an echo with Unicode to work.

Here’s the task code:

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [System.Text.Encoding]::UTF8
$pinfo = New-Object System.Diagnostics.ProcessStartInfo
$pinfo.FileName = "PowerShell"
$pinfo.RedirectStandardOutput = $true
$pinfo.RedirectStandardError = $true
$pinfo.UseShellExecute = $false
$pinfo.Arguments = "{arguments}"
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $pinfo
$p.Start()
$p.WaitForExit()
$p.StandardOutput.ReadToEnd() | Out-File {out_file} -NoNewline
$p.StandardError.ReadToEnd() | Out-File {err_file} -NoNewline
exit $p.ExitCode

Here it is in action: fmang@c86fd08

One issue remains: ipconfig, and presumably most cmd-ish programs won’t output UTF-8 even when the console is configured that way, but PowerShell will still try to decode them as UTF-8, causing corruptions. Given special characters didn’t work in the first place, we could specify that for old commands to work, one must explicity convert the output like ipconfig | Convert-From-OEM-To-Unicode, though I don’t know what PowerShell command can do the conversion.