Change Default Charset to utf-8

[yogurtearl]

If a response body doesn't specify a charset in the Content-Type, AsyncHttpClient defaults to ISO_8859_1.

For Content-Type: application/json without a charset specified, the default should be UTF-8, instead of ISO_8859_1.

Valid JSON is never encoded as ISO_8859_1
This would be a breaking change, so would have to be done at the appropriate time.

From the JSON RFC (https://tools.ietf.org/html/rfc7159) :

This document updates [RFC4627], which describes JSON and registers
   the media type "application/json".

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32.  The default
   encoding is UTF-8, and JSON texts that are encoded in UTF-8 are
   interoperable in the sense that they will be read successfully by the
   maximum number of implementations;

[slandelle]

ISO-8859-1 is the default charset when none is explicitly specified, as per HTTP/1.1 spec: http://www.w3.org/International/articles/http-charset/index

With AHC you can specify it either explicitly or in the Content-Type request header.
My first thought is that your problem is a user issue.
Then, if it could be proven that browsers can automagically figure out which encoding to use in such case, and such algorithm is pretty simple and efficient (doesn't involve trying to parse the whole content), we could give it a try.

[slandelle]

For Content-Type: application/json without a charset specified, the default should be UTF-8, instead of ISO_8859_1.

Do you have any reference to an RFC, or any browser based experimentation to share to support this statement?

[yogurtearl]

I haven't done browser experiments, but this is from the JSON RFC 7159 (https://tools.ietf.org/html/rfc7159) :

This document updates [RFC4627], which describes JSON and registers
   the media type "application/json".

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32.  The default
   encoding is UTF-8, and JSON texts that are encoded in UTF-8 are
   interoperable in the sense that they will be read successfully by the
   maximum number of implementations;

[slandelle]

I'm perfectly aware of the JSON spec. My point is it conflicts with the HTTP/1.1 one, as the default encoding is different.
I guess browser experiments is the only way to figure out what to do. I definitively don't want to implement a behavior different from what major browsers do (counting curl here).

When Content-Type is application/JSON and charset is not defined:
• do they use default browser's charset?
• do they use HTML page charset
• do they enforce UTF-8 because of ContentType?
• do they "fix" ContentType and add charset that wasn't specified?

Amending: Would some good soul be interested in running such investigations?

[bburket]

I can tell you that okhttp3 behaves like this (defaulting to UTF8), as does the .NET http client. In fact, I have never experienced application/json being encoded as 8859-1 until I used this HTTP client. Definitely burned me in production as well

[slandelle]

@bburket What do they do then? Do they simply fallback to UTF-8 for encoding String content, or do they also automatically add a charset attribute to the Content-Type header?

[bburket]

Well I believe we're talking the response here, so to say it appends to the header doesn't really makes sense.

You can see, for example, how okhttp3 deals with things here: https://github.com/square/okhttp/blob/c70e35802e984e8b9bf3fbf1479deb37c0bdfa43/okhttp/src/main/java/okhttp3/ResponseBody.java#L194

[slandelle]

Well I believe we're talking the response here

Nah, we're talking the request.

If you send a request with a Content-Type header equals to application/json, does it get "fixed" into application/json; charset=utf-8?

[bburket]

Maybe I'm confused? The very first line in the issue first filed is:

If a response body doesn't specify a charset in the Content-Type, AsyncHttpClient defaults to ISO_8859_1.

[slandelle]

Right, but the issue actually happens on both sides.

From RFC2068 3.7.1 (HTTP 1.1):

The "charset" parameter is used with some media types to define the character set (section 3.4) of the data. When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value.

Then, things get bloody in XHR spec:

The send(body) method must run these steps:
If state is not opened, throw an InvalidStateError exception.
If the send() flag is set, throw an InvalidStateError exception.
If the request method is GET or HEAD, set body to null.
If body is null, go to the next step.
Otherwise, let encoding and mimeType be null, and then follow these rules, switching on body:
Document
Set encoding to UTF-8.
Set mimeType to text/html if body is an HTML document, and to application/xml otherwise.
Then append ;charset=UTF-8 to mimeType.
Set request body to body, serialized, converted to Unicode, and utf-8 encoded.
BodyInit
If body is a string, set encoding to UTF-8.
Set request body and mimeType to the result of extracting body.
If mimeType is non-null and author request headers does not contain Content-Type, then append Content-Type/mimeType to author request headers.
Otherwise, if the header whose name is a byte-case-insensitive match for Content-Type in author request headers has a value that is a valid MIME type, which has a charset parameter whose value is not a byte-case-insensitive match for encoding, and encoding is not null, then set all the charset parameters whose value is not a byte-case-insensitive match for encoding of that header’s value to encoding.

which basically means that XHR only supports UTF-8

So well, that won't be the first time W3C contradicts itself...

Let's go with UTF-8 charset if none is specified...