Spawning AHC 2.0 w/ Netty 4 instances very fast leads to fd/thread/memory starvation

[dotta]

After updating the Play codebase to AHC 2.0-alpha9, I've started to experience issues with non terminating tests because of OutOfMemoryException. Until today, I wrongly assumed this was caused by the changes I made in AHC to support reactive streams, but the assumption turned out to be wrong. In fact, I can reproduce even by using AHC 2.0-alpha8, which doesn't include the reactive stream support.

Here is a link to the truly long thread dump demonstrating that AHC 2.0-alpha8 is leaking threads
https://gist.github.com/dotta/6e388962cf0d904e8170

This issue is currently preventing playframework/playframework#5082 to successfully build.

[dotta]

I'm going to try to create a test to reproduce the faulty behaviour.

[slandelle]

3 possibilities:

• Netty 4 issue
• AsyncHttpClient#close doesn't properly close all resources (the EventLoop, it seems)
• you're not properly closing, either because you're not closing AHC instances, or because you're providing some external resources, like a ThreadFactory, instead of letting AHC create its own. In this case you have to manage them yourself, AHC#close won't close them for you.

[dotta]

I'm speculating this is not a bad usage on the Play side because the problem started to manifest itself after updating AHC from 1.9.29 to 2.0.0-alpha8. But I don't have stronger evidence.

[dotta]

I just managed to reproduce the issue with AHC alone. I simply took a random test that creates a AsyncHttpClient client in the netty4 provider project, and surrounded the test code with a while(true), and you'll very quickly get a java.lang.OutOfMemoryError: unable to create new native thread, which is exactly the issue I'm seeing on the Play side.

[slandelle]

Could you please share your full sample?

[dotta]

Yep, I was going to do that, just needed time to properly package it. Also, the described issue only occurs with the netty4 extension. Here is a commit with the test to reproduce the problem dotta@67f9c32

[dotta]

Upgrading AHC to Netty 4.0.31 final doesn't help.

By the way, if the test I pushed doesn't fail for you, increase the number of iterations dotta@67f9c32#diff-64bc638cdb34d229f29fabdf5721d4cfR7

[slandelle]

AHC#close() calls at some point EventExecutorGroup#shutdownGracefully() which is a shortcut for EventExecutorGroup#shutdownGracefully(quietPeriod = 2, timeout = 15, SECONDS).

It means that an AHC instance don't close immediately, but give flying requests a last 2 seconds chance before actually trying to shut down.

In your test, you're spawning too fast.

Using those default values make sense for real world use cases. They might not for tests. I guess we could make those configurable.

[dotta]

Makes sense. And, indeed, if I change the call to shutdownGracefully with shutdown, then the test I wrote doesn't fail anymore. And that seems to be true also on the Play side, as I don't seem to be able to reproduce the OOM with AHC modified as described above.

@slandelle Thanks a lot for the super-quick follow up on this.

[dotta]

So, where should we add the configuration values, in AsyncHttpClientConfig?

[slandelle]

I guess so, yes. Under way.

[jfarcand]

@slandelle Since it's Netty specific, should it be in NettyAsyncHttpProviderConfig?

[dotta]

I guess so, yes. Under way.

That is awesome. Thanks!

[slandelle]

@jfarcand This is a Netty 4 only behavior, hence AHC2 where, hopefully, Netty 4+ will be the main target.