Enable string conversion in EUC-JP.

[YOCKOW]

Background: EUC-JP is not supported by OSS CoreFoundation, while it is supported by macOS Foundation Framework.
See #1016

This PR resolves the issue by calling ICU API if necessary.

[YOCKOW]

@swift-ci Please test

[YOCKOW]

@swift-ci Please test

[YOCKOW]

@parkera @jmschonfeld Thank you for reviewing.

In response to your feedback, I've made some changes:

• Made it clear that ICU.StringConverter doesn't support UTF-* and US-ASCII.
• Removed unnecessary nonisolated(unsafe) from the static property.
• Added comments in func _icuMakeStringFromBytes_impl to clarify the safety of passing the pointer.
• Let it delegate string conversion to ICU only if the encoding is EUC-JP and there's no FOUNDATION_FRAMEWORK (at this point).

[YOCKOW]

Please let me request reviews again as per changes above.

[parkera]

I think this looks good to me, but also want @jmschonfeld to chime in.

[jmschonfeld]

LGTM, thanks for putting this together!

[YOCKOW]

@swift-ci Please test

[jmschonfeld]

Thanks - that resolves the build failures, but I have a comment about one issue in the tests that I've discovered as well (this should hopefully be the last thing, thanks for bearing with me)

[jmschonfeld]

This test unfortunately fails in the FOUNDATION_FRAMEWORK configuration because Foundation.framework's implementation of this conversion replaces both UTF-16 scalars of the emoji with ? rather than replacing the entire grapheme cluster with a single ?. I suspect that maybe ICU is replacing only with a single ? - do you happen to know if that's intentional? If so, for now we might need to vary the expected value to be Sushi?? in FOUNDATION_FRAMEWORK vs. Sushi? in !FOUNDATION_FRAMEWORK to account for the discrepancy between Foundation.framework and ICU (assuming we think this result from ICU is indeed correct and not a bug in how we're calling ICU)

[YOCKOW]

Sorry for my overlooking again, but I'm not sure if we should implement conversion just like Foundation framework.
I mean the behavior of Foundation framework looks buggy because it doesn't seem to handle surrogate pairs correctly. (🍣 is non-BMP but consists of only one scalar U+1F363. Does that "buggy" behavior come from the historical reason that NSString has been represented as a sequence of UTF–16??)
Of course, we can imitate such behavior by replacing UCNV_FROM_U_CALLBACK_SUBSTITUTE with a custom callback function.
However, let me point out that current swift-foundation's implementation for .ascii is code-point-basis, not UTF-16-value-basis:

import Foundation

let sushiEmoji = "🍣"
print(sushiEmoji.data(using: .ascii, allowLossyConversion: true)!.count)
// -> Prints "1"

Options...?

• To keep consistent with current (e.g.).ascii implementation:
  • Divide the test case for FOUNDATION_FRAMEWORK and !FOUNDATION_FRAMEWORK.
• To make it compatible with Foundation framework's behavior:
  • Implement a custom callback for ucnv_setFromUCallBack.

[jmschonfeld]

Yeah I suspect this behavior comes from the fact that NSString is natively stored as UTF-16. For now, let's just divide the test case with a FOUNDATION_FRAMEWORK/!FOUNDATION_FRAMEWORK value for this expectation since I think enabling the conversion at all on non-Darwin is a good step even if the behavior is slightly different than Foundation.framework, and I can come back with a separate change to look into making this behavior the same across both platforms (while ensuring we don't break compatibility with existing clients).

[YOCKOW]

@jmschonfeld
Thank you for the advice. I revised the test cases.

(I didn't know CI didn't execute test cases with FOUNDATION_FRAMEWORK.)

[YOCKOW]

@swift-ci Please test

[jmschonfeld]

Thanks for all of your updates - looks great to me now!