Close connections periodically

[msfcolombo]

Problem

We have multiple VMs with AspNetCore running behind a load balancer. Clients reuse TCP connections as much as possible, and therefore VMs get "pinned" to a subset of clients. We often reach states where one or two VMs are using >90% of CPU resources, whereas most other VMs are <5%. This causes a noticeable degradation and is a recurrent event.

Proposal

One proposal is provide a setting that tells the maximum number of requests a given TCP connection can deliver.

Another is to allow closing a connection programatically. In this case, the HttpContext or other class should expose metrics such as number of requests delivered on that connection, although we can have a good estimate from the request payload itself (i.e. number of requests that a given IP address has performed) or we can simply throw dice and close connections on 0.1% of requests.

In either case, the developer should have ability tell if the closing of a connection will be immediate or graceful. In case of immediate termination of TCP connection, it's fine for ASP.NET to throw exceptions at I/O operations of previously existing requests.

In case of graceful termination, the service must only mark for closing, and stop processing new requests. Requests that exist before close initiated should be satisfied. In HTTP/11, the server can automatically return 503 responses for any request that arrives after close is initiated, until the last previously existing request is responded, after which the server can close the TCP socket. The server should also add Connection: close header as a hint to clients stop sending new requests. In HTTP/2 or HTTP/3, the server should send a GOAWAY frame and ignore new streams, as dictated by the RFC.

Workaround

The only workaround we know is to restart the AspNetCore application (or entire VM) periodically. However this is cumbersome to implement because we must restart only too-busy nodes, avoid simultaneous restarts of more than one node, and also avoid restarts of two distinct nodes too close to each other. If you have another workaround, please let us know!

Thanks
Fernando Colombo

[Tratcher]

Which server are you using in your app? Kestrel, IIS in-proc, IIS out-of-proc, HttpSys? The close the mechanics vary a little.

One proposal is provide a setting that tells the maximum number of requests a given TCP connection can deliver.
In this case, the HttpContext or other class should expose metrics such as number of requests delivered on that connection,

This kind of setting/information would only be possible on Kestrel. IIS/HttpSys don't expose the connection level semantics to us.

although we can have a good estimate from the request payload itself

?

(i.e. number of requests that a given IP address has performed)

A client is free to use multiple concurrent connections. Browsers do that by default. And when you're using a load balancer you don't get a 1:1 mapping between client connections and proxy connections, clients often end up sharing backend connections.

or we can simply throw dice and close connections on 0.1% of requests.

This is the approach I've seen implemented by customers.

Another is to allow closing a connection programatically.

This can be done in most places by setting the "Connection: Close" response header. There are a few cases where it's not honored like Kestrel Http/2 or IIS out-of-proc.

In either case, the developer should have ability tell if the closing of a connection will be immediate or graceful.

All closures are graceful unless you explicitly call the Abort APIs.

In HTTP/1, the server can automatically return 503 responses for any request that arrives after close is initiated

Http/1 connections only send one request at a time per connection unless you've enabled pipelining which is almost never used outside of benchmarks. A 503 mechanic shouldn't be needed.

In HTTP/2 or HTTP/3, the server should send a GOAWAY frame and ignore new streams, as dictated by the RFC.

Yes, it would refuse new streams and wait for old ones to drain.

[msfcolombo]

Chris, thanks for comments.

We use Kestrel. If you honor Connection: Close for Kestrel and HTTP/1.1, that fixes part of the problem. How about HTTP/2 and HTTP/3? You mention "Abort APIs". What APIs are these? Can you send link to documentation? Would these APIs work for HTTP/1.1 as well? Will these APIs cause a "graceful drain" of requests already read from the socket?

Few clarifications: 1) By HTTP/1 I've meant HTTP/1.1, not HTTP/1.0 - of course the later closes connection after each request; I have edited to make it more explicit. 2) Of course 1:1 mapping is hard to achieve; but we wish to avoid situations like 1:50 or worst for prolonged time. 3) This product is serving REST APIs; while browsers and proxies may use it, it's very uncommon. The common pattern is for clients to send many requests over a prolonged time, which makes "pinning" an often seen situation.

Thanks again!

[Tratcher]

Ok, some Kestrel specific details then. Kestrel honors you adding the Connection: close response header for Http/1.1. It will gracefully close the connection after the current request. This has not been implemented yet for HTTP/2 or 3. Kestrel will honor a GoAway but does not expose an API to trigger one.

HttpContext.Abort(): For Http/1.1 this forcefully aborts the connection and fails the in-flight request. For Http/2 & 3 it will cancel only the current request. There's a lower level API for Kestrel (IConnectionLifetimeFeature.Abort) that kills the connection no matter which protocol you use.

[msfcolombo]

Thanks. Please leave it open so we can track when HTTP/2 and HTTP/3 will honor Connection: close or provide a similar feature.

Also, there should be away of closing a TCP connection immediately in HTTP/2 and HTTP/3 as well (versus ending just the stream as it's probably doing today).

[Tratcher]

Also, there should be away of closing a TCP connection immediately in HTTP/2 and HTTP/3 as well

That's what IConnectionLifetimeFeature.Abort does.

Yes, I'll mark this as a feature request for Kestrel Http/2 & 3 graceful connection termination.