IdnMapping.GetAscii throws for some inputs in GlobalizationMode.Invariant

[stephentoub]

using System;
using System.Globalization;

class Program
{
    static void Main()
    {
        //Environment.SetEnvironmentVariable("DOTNET_SYSTEM_GLOBALIZATION_USENLS", "1");
        //Environment.SetEnvironmentVariable("DOTNET_SYSTEM_GLOBALIZATION_INVARIANT", "1");
        Console.WriteLine(new IdnMapping().GetAscii("\xD83D\xDE00"));
    }
}

This works fine with both NLS and ICU. But when GlobalizationMode.Invariant is enabled (uncomment the relevant line above), it throws an exception:

Unhandled exception. System.ArgumentException: Left to right characters may not be mixed with right to left characters in IDN labels. (Parameter 'unicode')
   at System.Globalization.IdnMapping.PunycodeEncode(String unicode)
   at System.Globalization.IdnMapping.GetAsciiInvariant(String unicode, Int32 index, Int32 count)
   at System.Globalization.IdnMapping.GetAscii(String unicode, Int32 index, Int32 count)
   at System.Globalization.IdnMapping.GetAscii(String unicode, Int32 index)
   at System.Globalization.IdnMapping.GetAscii(String unicode)

Tagging subscribers to this area: @tarekgh, @safern, @krwq
Notify danmosemsft if you want to be subscribed.

[tarekgh]

@stephentoub We don't promise any correct IDN behavior when enabling GlobalizationMode.Invariant. even it is not right anyone should depend on IDN behavior when GlobalizationMode.Invariant is enabled. We just got basic functionality working there and no promise to have any consistent result with disabling GlobalizationMode.Invariant. I suggest closing this one.

[stephentoub]

even if it is not right no one should depend on IDN behavior when GlobalizationMode.Invariant is enabled

We use this in higher-level APIs like HttpClient. This means that with invariant mode enabled, you'll get exceptions just by trying to access sites that cause IdnMapping to throw.

FYI, @richlander as you consider configurations for docker images and the tradeoffs involved.

[tarekgh]

I would argue it is good to get the exception at that time better than getting wrong behavior. I would worry about security more at that time.

[stephentoub]

My point is saying "We don't promise any correct IDN behavior when enabling GlobalizationMode.Invariant" means we also don't promise any correct behavior for anything higher in the stack, whether it be SslStream or HttpClient or anything else that uses it. I don't know how to reason about that. Can we be more specific about what will and won't work?

[tarekgh]

@stephentoub I understand your point. I was just commenting on what would be the choice between throwing and not throwing on the level of the IDN layer.

For the higher stack with the Invariant mode, we can have the choice to change the Invariant mode behavior when the system already has NLS/ICU. This will be the case always on Windows and MacOS and only be broken on Linux images not having ICU installed. the concern with that is we are not fully solving the problem and we are losing the consistency promise when enabling Invariant. Note, I am talking about the IDN only and not the whole globalization.

[stephentoub]

I understand your point.

👍

[tarekgh]

I am moving this to 6.0 release as this is not a new behavior for invariant mode. we can revisit later to enhance Invariant mode if needed.