NullPointerException

[aqingsao]

I've been using async-http-client for a long time, it works fine in many cases even at a high QPS of 100K.
But recently some errors happen when calls to a specific api(nothing special, just a POST http with Content-Type: multipart/form-data), lots of exceptions are thrown at a low QPS of 800, and below is the stack trace:

`java.lang.NullPointerException: null
at org.asynchttpclient.netty.channel.Channels.getAttribute(Channels.java:31) ~[async-http-client-2.12.2.jar!/:na]
at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:467) ~[async-http-client-2.12.2.jar!/:na]
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.exceptionCaught(AsyncHttpClientHandler.java:201) ~[async-http-client-2.12.2.jar!/:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:264) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:138) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:235) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]

Any idea of what can I do?

[TomGranot]

Hey @aqingsao - new maintainer here, still learning the ropes.

Can you show the exact code you used + mention some specs? Might help the investigation.

[aqingsao]

Hi TomGranot,
Glad to see u and welcome.
I use AsyncHttpClient to build our load test platform, it works fine in most cases.
But in a few cases, some exceptions(NullPointerException) are thrown at about 0.1% rate or even 30% in a very special case.
Actually these case are just normal http POST requests of type 'form-data'.

Some codes are pasted in below but I don't think it provides any clues.

this.builder.setUserAgent("xxx"); this.builder.setThreadFactory(new CustomThreadFactory("netty-client")); this.builder.setIoThreadsCount(Math.max(Runtime.getRuntime().availableProcessors() / 2, 2)); this.builder.setMaxConnections(MAX_CONNECTIONS); this.builder.setMaxConnectionsPerHost(MAX_CONNECTIONS_PER_HOST); this.builder.setMaxRequestRetry(0); try { this.builder.setSslContext(SslContextBuilder.forClient() .sslProvider(SslProvider.OPENSSL) .trustManager(InsecureTrustManagerFactory.INSTANCE).build() ); } catch (SSLException e) { log.error("Failed to set SSL config", e); } this.builder.setCookieStore(null); new DefaultAsyncHttpClient(this.builder.build());

httpClient.executeRequest(request, new ProgressHandler(request, callback)).toCompletableFuture();

[rachid-o]

I think I have the same issue. We have a unit test where we simulate a "connection reset by peer" which worked fine on async-http-client version 2.12.1
But when updating to version 2.12.2 we got this NPE:

 java.lang.NullPointerException
	at org.asynchttpclient.netty.channel.Channels.getAttribute(Channels.java:31)
	at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:467)
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.exceptionCaught(AsyncHttpClientHandler.java:201)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:264)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:138)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:235)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
	at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311)
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

For what I've seen in NettyRequestSender.abort() there is this line:

Object attribute = Channels.getAttribute(future.channel());

and future.channel() == null

[rachid-o]

Some extra information.
Before suspecting async-http-client we downgraded Netty to 4.1.53.final, but then we still got the same problem.

We could reproduce it at our side with the following unit test which uses WireMock (Kotlin code):

private val httpClient: AsyncHttpClient = DefaultAsyncHttpClient(
    DefaultAsyncHttpClientConfig.Builder()
        .setRequestTimeout(2000) // Overwrite the default of 60 secs so that unit test fail faster on timeouts
        .build()
)
private val wireMockServer: WireMockServer = WireMockServer(wireMockConfig().dynamicPort())

@Test
fun `Should throw IOException when connection is reset by peer`() {
    wireMockServer.stubFor(
        get(urlEqualTo("/test"))
            .willReturn(
                aResponse()
                    .withFault(Fault.CONNECTION_RESET_BY_PEER)
            )
    )
    val testSubscriber = TestSubscriber<Response>()
    response().subscribe(testSubscriber)
    testSubscriber.awaitTerminalEvent()
    testSubscriber.assertError(IOException::class.java)
}

I took a quick look in the source code, but couldn't directly find a way to create a test case simulating a "connection reset by peer" in this repo.
@TomGranot let me know if I can assist. Maybe you or someone else can push me in the right direction so that I can create a failing unit test.

[dzharikhin]

Hi, I extracted resetting server logic from the tests I have. Here's simple test

public class AsyncHttpClientTest {

    private String resettingServerAddress;

    @Before
    public void setUp() {
        resettingServerAddress = createResettingServer();
    }

    @Test
    public void testAsyncHttpClient() throws ExecutionException, InterruptedException {
        try {
            new DefaultAsyncHttpClient().executeRequest(new org.asynchttpclient.RequestBuilder("GET").setUrl(resettingServerAddress)).get();
        } catch (ExecutionException e) {
            var ex = e.getCause();
            assertTrue(ex instanceof SocketException);
            assertTrue(ex.getMessage().equalsIgnoreCase("connection reset"));
        }
    }

    private static String createResettingServer() {
        return createServer(sock -> {
            try (Socket socket = sock) {
                socket.setSoLinger(true, 0);
                var inputStream = socket.getInputStream();
                //to not eliminate read
                OutputStream.nullOutputStream().write(startRead(inputStream));
            }
        });
    }

    private static String createServer(MoreFunctionalInterfaces.FailableConsumer<Socket, Exception> handler) {
        Exchanger<Integer> portHolder = new Exchanger<>();
        var t = new Thread(() -> {
            try (ServerSocket ss = new ServerSocket(0)) {
                portHolder.exchange(ss.getLocalPort());
                while (true) {
                    handler.accept(ss.accept());
                }
            } catch (Exception e) {
                if (e instanceof InterruptedException) {
                    Thread.currentThread().interrupt();
                }
                throw new RuntimeException(e);
            }
        });
        t.setDaemon(true);
        t.start();
        return tryGetAddress(portHolder);
    }

    private static String tryGetAddress(Exchanger<Integer> portHolder) {
        try {
            return "http://localhost:" + portHolder.exchange(0);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    private static byte[] startRead(InputStream inputStream) throws IOException {
        byte[] buffer = new byte[4];
        int length = inputStream.read(buffer);
        return Arrays.copyOf(buffer, length);
    }
}

[rachid-o]

That certainly helps @dzharikhin!
I molded your code into a fork in this branch: master...rachidbm:2.12.2
There I see the NPE, which I don't see when running the same test on master...rachidbm:2.12.1

[rachid-o]

Thnx for merging @TomGranot, this issue can be closed now.

FWIW, when using the new version of async-http-client (2.12.3) the issue is solved in our project.

[TomGranot]

@rachidbm Excellent, closing this as well.