[V1] Ensure there's enough memory for all warmup shapes

[joerunde]

The warmup shapes provided at startup place a hard cap on the number and length of sequences in a batch that will be scheduled. After we load the model onto the card, we should ensure that there is enough available memory to run all possible shaped batches. This could also respect the --gpu-memory-utilization parameter, so that users can set a configurable buffer.

This would likely require implementing memory profiling so we can calculate how much memory a given batch size will take at inference

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!