CONNECT request is sent with default AHC user-agent, not with the on from RequestBuilder

[insticore]

The new CONNECT request is made by NettyRequestFactory, where User-Agent is set from AsyncHttpClientConfig, if I correctly understand.
So in result the user-agent differs from the following requests.

Please, is it possible to fix?

This is captured by Wireshark, the first request CONNECT and the headers are:
accept: /.
user-agent: AHC/2.1

[slandelle]

I no longer maintain this project (actually looking for new maintainers).
Contributing a fix would be very much welcomed.

[TomGranot]

@insticore coming a bit late for the party, not sure if it's still relevant. I put my hat in for maintaining the repo, so figured it won't hurt to start digging through issues anyways.

In the meanwhile - and before I open a PR to fix - let me try and see whether I understand the issue completely:

You're seeing a user-agent mismatch between... what and what exactly? I'll try and reproduce and inspect the packets, just to make sure I get the same thing.

Consider the following, simple usage example taken from one of the examples in this repo:

package org.asynchttpclient.example.completable;
import org.asynchttpclient.AsyncHttpClient;
import org.asynchttpclient.Response;
import java.io.IOException;
import static org.asynchttpclient.Dsl.asyncHttpClient;

public class CompletableFutures {
  public static void main(String[] args) throws IOException {
    try (AsyncHttpClient asyncHttpClient = asyncHttpClient()) {
      asyncHttpClient
              .prepareGet("http://www.reddit.com/")
              .execute()
              .toCompletableFuture()
              .thenApply(Response::getResponseBody)
              .thenAccept(System.out::println)
              .join();
    }
  }
}

Note that I swapped http://www.example.com with http://www.reddit.com, which is 301'd.

Looking at WireShark, I'm seeing the following outbound packet:

Which corresponds to what you mentioned. So I can reproduce, but I'm not sure I understand what a desired outcome on your side would be. Can you please be a bit more specific?

[TomGranot]

@insticore Just pinging again to see whether you can clarify a bit more. No rush:)

[insticore]

@TomGranot hi there, thanks a lot for keeping this great project alive!
The issue here is that I need to use some specific user agent for each request by adding it as an 'User-Agent' header like here
Dsl.get(<url>).addHeader("user-agent", <custom UA>).
But when I use HTTPS connection, there are two HTTP requests sent, the first one is a CONNECT with user-agent not from RequestBuilder, but the one from AsyncHttpClientConfig ("AHC/2.1" by default). And only for the second request (GET one) my custom user-agent is set correctly.

[TomGranot]

@insticore today's a rough one, but will take a look tomorrow evening. If you could ping me if i forget that would be great;)

[insticore]

@TomGranot hello, just asking if there any news for this issue
And also found a screenshot I did for a request with incorrect user-agent
.
So here is the first CONNECT request with default "AHC/2.1", and the next requests have custom user-agent I set within the code.

[TomGranot]

Just to clarify for anybody reading this at a later point in time - this does not affect you if you don't have a proxy server in the way. In @insticore's case it looks like there's an SSL proxy that's sitting between them and the internet, causing the normal flow of HTTPS to be a bit different. Instead of:

1. HTTPS request to original host

We now have:

1. HTTP request to proxy
2. HTTPS requests from proxy to original host

What @insticore is saying is that in request no. 1 he's seeing the wrong user-agent. What I assume happens is that the interceptor in charge of handling proxy requests (and more specifically, proxy authentication requests) uses the AsyncHttpClient default user-agent. This means that the CONNECT http request sent to the proxy does not "receive" the correct user agent string as you would expect it to.

A bit of digging first - I think that the culprit is this piece here, that uses the default user-agent if no user-agent is found. A small modification of the test in here (basically just adding a setHeader("user-agent","TomUserAgent") to the outgoing request) reproduces the behaviour you're mentioning (relevant lines emphasised):

Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
CONNECT localhost:61329 HTTP/1.1
host: localhost:61329
proxy-authorization: Basic am9obmRvZTpwYXNz
accept: */*
---------------------------->>>>>>> user-agent: AHC/2.1

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Sat, 07 Nov 2020 18:58:03 GMT
Content-Length: 0
Server: Jetty(9.4.18.v20190429)

Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET /foo/bar HTTP/1.1
--------------------------->>>>>>> user-agent: TomUserAgent
host: localhost:61329
accept: */*

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Sat, 07 Nov 2020 18:58:03 GMT
Content-Type: text/html;charset=utf-8
X-User-Agent: TomUserAgent
X-Accept: */*
X-Host: localhost:61329
X-pathInfo: /foo/bar
X-KEEP-ALIVE: 127.0.0.1:61337
Transfer-Encoding: chunked
Server: Jetty(9.4.18.v20190429)

As you mentioned the problem is indeed in NettyRequestFactory, and more specifically here - we can simply set the header for USER_AGENT to be the same as in the request (very similar to what we do with the authorization header) and voila:

Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
CONNECT localhost:65315 HTTP/1.1
--------------------------->>>>>>> user-agent: TomUserAgent
host: localhost:65315
proxy-authorization: Basic am9obmRvZTpwYXNz
accept: */*

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Sat, 07 Nov 2020 20:18:56 GMT
Content-Length: 0
Server: Jetty(9.4.18.v20190429)

Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET /foo/bar HTTP/1.1
--------------------------->>>>>>> user-agent: TomUserAgent
host: localhost:65315
accept: */*

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Sat, 07 Nov 2020 20:18:56 GMT
Content-Type: text/html;charset=utf-8
X-User-Agent: TomUserAgent
X-Accept: */*
X-Host: localhost:65315
X-pathInfo: /foo/bar
X-KEEP-ALIVE: 127.0.0.1:65319
Transfer-Encoding: chunked
Server: Jetty(9.4.18.v20190429)

PR: #1742

[TomGranot]

@insticore Merged into master, will be released in 2.12.3.