Improve Http.sys allocation profile and performance

[davidfowl]

AB#1240693

This isn't urgent in anyway but I wanted to keep track of the low hanging performance fruit in HttpSys server:

Http.sys optimizations

General

• Feature collection generation similar/uses kestrels (Unify FeatureCollection and Header code generation between server implementations #31323)
• Headers generation shares with Kestrel (Unify FeatureCollection and Header code generation between server implementations #31323)

Accept loop

• Use a single preallocated native overlapped per server (dispose after the accept loop is over)
• This means getting rid of SafeNativeOverlapped (or making it a struct)
• Make the NativeRequestContext a struct, it's basically a wrapper around a byte with helpers to extra data from a blob of memory.
  • Another option would be to merge it into the feature collection implementation (similar to IIS)
• Allocate a single feature collection object that implements the request and response contract. Today we have 2 objects maybe out of legacy?

Bodies

• Synchronous read can be implemented by pinning the user buffer (or we can throw like we do in Kestrel)
• Implement span overloads (these are missing)
• Remove Begin/End implementations, and use Task.ToApm helpers to implement Begin/End over
  • The Task/ValueTask based overloads (the Memory and Span overloads should be the only implementation)
  • Remove IAsyncResult implementations
• Implement pattern from IIS implementation IValueTaskSource based implementations of IO
  • Single object re-used for IO (each request body), no overlapping operations allowed.
  • Reuse pre-allocated overlapped during reads and writes
• Potentially use memory pool for output
• This removes the need to pin user buffers and avoid fragmentation
• May result in more copies (user buffer-> pinned buffer)

Client cert loader

• TBD

[wtgodbe]

Will get to this once I'm done with teamCity - I'll loop in @davidfowl, @sebastienros and @Tratcher once I get started

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.