[V1] batch scheduling of simultaneously arriving requests

[yannicks1]

V1 Scheduler: Batch simultaneously arriving requests together

The current V1 vllm-spyre scheduler does not schedule request, which have been submitted simultaneously, in the same batch.
Note: The V0 vllm-spyre scheduler did schedule these requests together.

How to reproduce

alter examples/offline_inference_spyre.py to use batch size 4 and submit 4 prompts simultaneously:

os.environ['VLLM_SPYRE_WARMUP_BATCH_SIZES'] = '4'
.
.
.
prompts = [prompt1, prompt1, prompt1, prompt1]

V1 yields (export VLLM_USE_V1=1):

SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
Processed prompts:   0%|                                                                                          | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s][spyre_model_runner:execute_model] t_token: 231.08ms
[spyre_model_runner:execute_model] t_token: 67.17ms
[spyre_model_runner:execute_model] t_token: 65.63ms
[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
Processed prompts:  25%|████████████████████                                                            | 1/4 [00:00<00:01,  2.75it/s, est. speed input: 134.60 toks/s, output: 8.24 toks/s][spyre_model_runner:execute_model] t_token: 171.95ms
[spyre_model_runner:execute_model] t_token: 75.90ms
[spyre_model_runner:execute_model] t_token: 62.83ms
Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  5.92it/s, est. speed input: 290.16 toks/s, output: 17.76 toks/s]
Time elaspsed for 3 tokens is 0.69 sec

V0 yields (export VLLM_USE_V1=0):

[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
[SpyreModelRunner] INFO: Padding request of length 49 tokens to 64 tokens.
[spyre_model_runner:execute_model] t_token: 166.63ms
[spyre_model_runner:execute_model] t_token: 57.04ms
[spyre_model_runner:execute_model] t_token: 57.95ms
Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 14.03it/s, est. speed input: 687.52 toks/s, output: 42.09 toks/s]
Time elaspsed for 3 tokens is 0.29 sec

Clearly one can see that the V1 scheduler schedules just the first sequence (padded with 3 empty sequences) and after these decodes have finished, the remaining 3 sequences (padded with 1 empty sequence). The V0 scheduler batches all 4 sequences in a single batch, which is the desired behavior.

Note: experiments run on CPU.

[yannicks1]

I strongly believe this can be easily solved by adding a (very short) delay. I just observed in an experiment with V1 that it once scheduled all sequences in the first batch, and in the next (identical) experiment produced above behavior again. I checked whether we could use this contribution of our team to introduce a delay, but they don't seem to have added it in V1 (yet).

[joerunde]

It should be possible to add that logic to our own scheduler as well, and not require an upstream contribution for now, if we wanted to go that route

[joerunde]

@yannicks1 I think this bit of code from the new-ish v1 oracle indicates that they won't be supporting this in the v1 scheduler: https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py#L1532-L1535

IIUC, the reccomend_to_remove=True flag is a soft deprecation telling users to remove that cli flag from their configurations for V1