☂️ `String.init(data:encoding:)`/`String.data(using:)` regression.

[YOCKOW]

This is an umbrella issue that is tracking regression of String.init(data:encoding:)/String.data(using:).

Data → String

• Non-lossy ASCII: Cannot instantiate String with Data with encoding nonLossyASCII #929

String → Data

• Shift_JIS: String.data(using:) does not support Shift-JIS. #925
• ISO-2022-JP: Regression: String.data(using: .iso2022JP) unexpectedly returns nil on non-Darwin. #1014
• EUC-JP: Discrepancy: String.data(using: .japaneseEUC) returns nil on non-Darwin. #1016
• CP437: Swift should support kCFStringEncodingDOSLatinUS on Linux #1029

[Frizlab]

Should this one be part of this or is the current behavior considered the correct one?

[YOCKOW]

Should this one be part of this or is the current behavior considered the correct one?

Thank you. I added it.
But I'm not sure whether or not it should be reopened because Darwin's behavior seems to have also changed.

[YOCKOW]

@parkera
I think these are signals which imply that the decision not to support string encodings we are still using is a little slapdash.
On the other hand, we have to consider what @omochi said in #925:

String.data is defined in FoundationEssentials, not in FoundationInternationalization, so ICU cannot be used.

IMHO, options are:

• To lower data(using:) onto FoundationInternationalization to use ICU.
• To implement string converters in pure-Swift.
• To ignore these signals.

[parkera]

We need data(using:) for non-internationalization related APIs, which is why it is implemented in Essentials. I don't quite understand option 1 here, because FoundationInternationalization is layered above Essentials.

We can move more encoding conversions to be implemented in Swift over time. However, if importing the full Foundation module on Linux we also might be able to support the larger set of encodings by FoundationEssentials "upcalling" into swift-corelibs-foundation's CoreFoundation string encoding conversion.

[YOCKOW]

I don't quite understand option 1 here, because FoundationInternationalization is layered above Essentials.

I'm sorry, but I misworded due to my cultural background. ("lower" → "lift"?)
I meant option 1 was to let data(using:) use directly ICU in any way.

if importing the full Foundation module on Linux we also might be able to support the larger set of encodings by FoundationEssentials "upcalling" into swift-corelibs-foundation's CoreFoundation string encoding conversion.

Is this some kind of "future direction"?
I thought current available stopgap workaround was to call CF APIs directly.

[parkera]

Take a look at the technique we use in Calendar_Cache.swift for an example. When on Darwin (where this is all one module) or when on non-Darwin and a callback has been registered, we can allow the lower-level Calendar(identifier:) API to support identifiers that only work when FoundationInternationalization is loaded.

[YOCKOW]

I didn't know such technique. I appreciate your instruction. I've been illuminated.
Thank you.

[YOCKOW]

Sorry for letting this sit for a while.
This seems to be resolved thanks to #1217 and swiftlang/swift-corelibs-foundation#5194.
Nonetheless, is it legitimate to keep this issue open because we'd better graduate from corelibs-foundation and string↔︎data conversions should be implemented in Swift (or should wrap foundation-icu) in the future?

[parkera]

I'd prefer to close this issue since it is resolved. We can have individual issues to track reimplementing encoding in Swift, where appropriate.