Add config option to eject channels from connection pool after 5xx error response

[kstokoz]

Background

In AWS standard service architecture have ELB with several backend nodes. In case of globally load balanced service you will have same in several regions. In case one of the regions has problems in AWS there is an option to fail-over. During fail-over IPs from problematic regions no longer distributed to clients in DNS lookup.

Unfortunately if you are using AHC with connection pooling in case of fail-over requests will continue to be routed to region which has a problem. All request will have 5xx error codes.

Issue

To be able to recover with AHC from such situation is would be great to have a config option not to add such problematic channels back to the pool. So that when new connection required DNS lookup will get healthy region.

For 1.8.x i was able to to do it easily as NettyResponseFuture had a httpResponse object inside

protected final boolean getKeepAlive() {
    boolean is5xxError = (httpResponse != null && httpResponse.getStatus().getCode() >= 500);
    return keepAlive && !is5xxError;
} 

in 1.9.x there was a major change in this area and it is no longer easy to do this change.

[slandelle]

When such a disaster occurs, don't you think you should be flushing the whole cache partition instead?
https://github.com/AsyncHttpClient/async-http-client/blob/1.9.x/src/main/java/com/ning/http/client/providers/netty/NettyAsyncHttpProvider.java#L94

[kstokoz]

I agree that this issue request is not providing ideal solution to a problem and looks more like workaround.
Flushing partition looks interesting, but on client side it is really hard to detect if whole regions is affected or it is one of 503 error. As you can imagine flushing for every 5xx error is also not acceptable in heavy loaded environments.

To be able to decide to flush pool or not, we would need to have data, what would require adding state on client.

[slandelle]

@kstokoz I'm still puzzled by what you describe (not an AWS expert): if the region has a problem, why do you get 503 responses? Why doesn't AWS close the connections?

Having a way to pass a strategy for deciding to pool or close a connection is completely feasible but I'd first like to really understand what happens and if there's not a way to get the proper behavior instead of having an ugly down-flow workaround. IMHO, the core issue is an AWS one. As you're paying for their service, did you get in touch with their support and ask them why connections to a crashed region don't get closed?

[kstokoz]

Imagine you have a multiregional service. In every region you normally have load balancer(ELB), several application servers and some shared resource (DB).

If for whatever reason DB goes down, application servers immediately report themselves as unhealthy to ELB. In this case ELB remove all servers from rotation and you end up with ELB without back end nodes. ELB in this situation switch to 503 mode, for every request return 503.

At this point if you have a DNS fail over setup, unhealthy region will be removed from DNS resolution. All newly established connections will go into other regions.

The problem though exists for already existing to old unhealthy ELB connections. Right now 503 is a valid response code so AHC will continue using existing channel.

Rethinking above described scenario and rereading http spec it seems correct solution will be that in such sition ELB should return 503 and header forcing client to close connection.

We've contacted amazon already regarding such situations, let see what they say.

If ELB would be able to notify client in such way it would have same effect as initially was requested in this issue.

[slandelle]

IMHO, the proper solution would be to have the ELB terminate all the connections linked to a crashed region. This way, you'd never get those 503 failures.

A partial solution would indeed to have the ELB replies 503 + Connection: close header, but then, only the connection for this request wouldn't be offered back to the pool. What about all the other ones? You'd still get 503 for quite some time, until all connections to the crashed region have been used and removed.

Then, i would also makes sense to introduce a configurable strategy that let the user decide when a connection can be offered to the pool. For example, there's a configuration parameter that let one disable HTTPS connections pooling. This could be turned into an abstraction.

[kstokoz]

I agree proper solution would be to close all connection on ELB side in this situation.

Fully disable connection pooling is also not a solution in this case as well. Our stats show that to establish tcp connection on avg takes ~100ms + ssl handshake overhead +100-200ms, so this is very expensive time-wise (in case of https CPU-wise) operation in heavy loaded environment.

Having ELB return 503 with Connection:close header is better than what is happening now.

In our situation we do not expect in any normal circumstances to get from ELB connection close header. If this would happen we would probably flush connections partition related to this remote host to prevent further failures of request,

So strategy to make such decisions would be very helpful.