System.InvalidOperationException: NativeOverlapped cannot be reused for multiple operations after .NET 6 to .NET 8 upgrade #54251
Description
Is there an existing issue for this?
- I have searched the existing issues
Describe the bug
After migration from .NET 6 to .NET 8 we started seeing multiple .NET Runtime crashes with below callstacks. The callstack suggests a problem with IO completion and async operations. It does not point to any application code, so I am seeking for help. I have memory dumps of both crash scenarios.
Windows Server 2022
HTTP.sys ASP.NET Core
CoreCLR Version: 8.0.123.58001
.NET Version: 8.0.1
Expected Behavior
No crash.
Steps To Reproduce
No response
Exceptions (if any)
Application: foo
CoreCLR Version: 8.0.123.58001
.NET Version: 8.0.1
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException: NativeOverlapped cannot be reused for multiple operations.
at System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading.ThreadPoolTypedWorkItemQueue`2.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Application: foo
CoreCLR Version: 8.0.123.58001
.NET Version: 8.0.1
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException: Operation is not valid due to the current state of the object.
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.OnCompleted(Action`1 continuation, Object state, Int16 token, ValueTaskSourceOnCompletedFlags flags)
at Microsoft.AspNetCore.Server.HttpSys.AsyncAcceptContext.OnCompleted(Action`1 continuation, Object state, Int16 token, ValueTaskSourceOnCompletedFlags flags)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.System.Runtime.CompilerServices.IStateMachineBoxAwareAwaiter.AwaitUnsafeOnCompleted(IAsyncStateMachineBox box)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AwaitUnsafeOnCompleted[TAwaiter](TAwaiter& awaiter, IAsyncStateMachineBox box)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Application: foo
CoreCLR Version: 8.0.123.58001
.NET Version: 8.0.1
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException: 'overlapped' has already been freed.
at System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading.ThreadPoolTypedWorkItemQueue`2.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
.NET Version
8.0.1
Anything else?
In the second crash we see _mrvts is in completed state with cached request context. I wonder if _mrvts.SetResult is throwing before returning, which is then caught by the catch block which calls _mrvts.SetException and throws because _mrvts is already in completed state. Or maybe IO completion is called twice on the same _mrvts.
