Skip to content

backport 8.0 - http.sys accept loop - mitigate against break due to possible conflicting IO callbacks #54437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 12, 2024

Conversation

mgravell
Copy link
Contributor

@mgravell mgravell commented Mar 8, 2024

backport #54368 to .NET 8

Context: #54251

System.InvalidOperationException: NativeOverlapped cannot be reused for multiple operations after .NET 6 to .NET 8 upgrade

This manifests as a fatal exception in the http.sys "accept" loop, which can either:

  • fault in purely managed code, stopping the message pump (MessagePump.ExecuteAsync)
  • fault during an IOCP callback, potentially killing the process

Description

Apply mitigation and logging

Fixes #54251

Customer Impact

without fix, http.sys web-server stops taking traffic (best case) or segfaults

Regression?

  • Yes
  • No

believed to be a regression since .net 6; scenario is hard to repro reliably

Risk

  • High
  • Medium
  • Low

This change is "fresh", and hasn't been validated in nightly (or even merged to "main"); the technical changes are minor (logging, improved error handling), and the competing behaviour is catastrophic.

mgravell added 4 commits March 8, 2024 16:44
1. handle MRTVS cascading fault breaking the accept loop
2. log any expectation failures

# Conflicts:
#	src/Servers/HttpSys/src/AsyncAcceptContext.cs
…itching

- lock new critical log messages behind app-context switch
@ghost ghost added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Mar 8, 2024
@mgravell mgravell changed the base branch from main to release/8.0 March 8, 2024 16:58
@mgravell mgravell added Servicing-consider Shiproom approval is required for the issue feature-httpsys labels Mar 8, 2024
@mgravell mgravell marked this pull request as ready for review March 8, 2024 17:01
@amcasey amcasey removed the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Mar 8, 2024
@mgravell mgravell added Servicing-approved Shiproom has approved the issue and removed Servicing-consider Shiproom approval is required for the issue labels Mar 11, 2024
@mgravell
Copy link
Contributor Author

approved by servicing tactics 8th /cc @wtgodbe

@mgravell
Copy link
Contributor Author

@adityamandaleeka @Tratcher @halter73 @JamesNK code review approval needed ASAP

var value = Interlocked.Exchange(ref _expectedCompletionCount, 1); // should have been 0
if (value != 0)
{
if (_logExpectationFailures)
Copy link
Member

@halter73 halter73 Mar 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why even gate this behind a flag? I doubt many people will know to turn it on, and if these are failing commonly, wouldn't we want to hear about it? You could disable the HttpSysListener logger if it's too noisy.

Or at least make it opt-out. This is another case where being able to filter logs by event ID would be nice. dotnet/runtime#49924

{
Log.AcceptSetExpectationMismatch(_logger, value);
}
Debug.Assert(false, nameof(SetExpectCompletion)); // fail hard in debug
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:

Suggested change
Debug.Assert(false, nameof(SetExpectCompletion)); // fail hard in debug
Debug.Fail($"nameof(SetExpectCompletion) - {value}"); // fail hard in debug

@wtgodbe wtgodbe merged commit ece10b1 into release/8.0 Mar 12, 2024
@wtgodbe wtgodbe deleted the marc/backport-http-sys branch March 12, 2024 17:12
@dotnet-policy-service dotnet-policy-service bot added this to the 8.0.4 milestone Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-httpsys Servicing-approved Shiproom has approved the issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

System.InvalidOperationException: NativeOverlapped cannot be reused for multiple operations after .NET 6 to .NET 8 upgrade
4 participants