-
Notifications
You must be signed in to change notification settings - Fork 869
Closed
Labels
bugThis issue is a bug.This issue is a bug.module/sdk-customresponse-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 7 days.Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Description
Describe the bug
var response = await _client.InvokeModelWithResponseStreamAsync(request);
//there are two ways to get the streaming events. one is enumeration.
foreach (var e in response.Body)
{
}
//the other is setting up events + StartProcessing():
response.Body.ChunkReceived += (sender, args) => { /* handle event*/};
response.Body.StartProcessing();
//(in this 2nd way there does not seem to be a natural way to know when it's finished?
//both techniques end up in this blocking NetworkStream.Read() call:
//https://github.com/aws/aws-sdk-net/blob/976e5b161fc1d41b1b8e57e47597c834ba3c40a4/sdk/src/Core/Amazon.Runtime/EventStreams/Internal/EventStream.cs#L310
This is unfortunate. The threads that are supposed to handle new incoming requests for my server, are now blocked on very long running blocking network calls. (AI model responses are easily 30s, (spread out over many events)).
Expected Behavior
I would expect a way that allows me to use async enumeration:
await foreach(var event in response.EnumerateEventsAsync())
{
//handle event.
}
EnumerateEventsAsync would under the hood not use blocking io, leaving my servers
threads available to do server work, and come back to this network request only when
data is there.
Current Behavior
server threads being blocked.
Reproduction Steps
see code snippet in description
Possible Solution
No response
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
Targeted .NET Platform
net8
Operating System and version
all
Metadata
Metadata
Assignees
Labels
bugThis issue is a bug.This issue is a bug.module/sdk-customresponse-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 7 days.Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.