Consider to support delay scheduling in Gateway

[Jeffwan]

🚀 Feature Description and Motivation

Delay scheduling request to avoid over-assignment to some inference engines. We actually have the discussion on the push or pull based solutions. This would be a compliment of the push solution.

Use Case

No response

Proposed Solution

No response

[Jeffwan]

part of #99

[Jeffwan]

this is to avoid over-assignment

[varungup90]

@zhangjyr reassigned to you, since you have already started working on it.

[Jeffwan]

@zhangjyr I remember our review feedback earlier is to have decoupled PR from the cache refactoring. cache refacor PR was merged, delay scheduling PR was not out right? I think it's not urgent for v0.3.0 release. If that one need more time on testing and performance evaluation, I think leave it to v0.4.0 makes more sense. Feel free to let us know your thoughts.

[zhangjyr]

In fact. My SLO-based routing policy introduced a new abstraction called QueueRouter, which allows reordering requests based on the queue type. The QueueRouter is a framework-level router that bridges between the ordering of requests (queue) and the routing of requests. So the QueueRouter can be regarded as a solution for this ticket. It provides the capability to implement a delayed router, and the SLO router is an example of a delayed router.

[Jeffwan]

@zhangjyr few questions,

• SLO-based routing policy is associated with this issue [RFC]: Load-aware pattern-based routing policy with profile support #642?
• If other routing policies need delay scheduling as a common capability to leverage, is it possible? Or it only works with SLO-based routing policy? I am trying to understand whether we need to leave this story. Seems the scope is a little bit different.