Skip to content

There is no way to get bedrock streaming api responses using async io. #3360

@lucasmeijer

Description

@lucasmeijer

Describe the bug

       var response = await _client.InvokeModelWithResponseStreamAsync(request);
        
        //there are two ways to get the streaming events.  one is enumeration.
        foreach (var e in response.Body)
        {
        }
        
        //the other is setting up events + StartProcessing():
        response.Body.ChunkReceived += (sender, args) => { /* handle event*/};
        response.Body.StartProcessing();
        //(in this 2nd way there does not seem to be a natural way to know when it's finished?
        
        //both techniques end up in this blocking NetworkStream.Read() call:
        //https://github.com/aws/aws-sdk-net/blob/976e5b161fc1d41b1b8e57e47597c834ba3c40a4/sdk/src/Core/Amazon.Runtime/EventStreams/Internal/EventStream.cs#L310

This is unfortunate. The threads that are supposed to handle new incoming requests for my server, are now blocked on very long running blocking network calls. (AI model responses are easily 30s, (spread out over many events)).

Expected Behavior

I would expect a way that allows me to use async enumeration:

await foreach(var event in response.EnumerateEventsAsync())
{
    //handle event.
}

EnumerateEventsAsync would under the hood not use blocking io, leaving my servers
threads available to do server work, and come back to this network request only when
data is there.

Current Behavior

server threads being blocked.

Reproduction Steps

see code snippet in description

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Targeted .NET Platform

net8

Operating System and version

all

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.module/sdk-customresponse-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 7 days.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions