[Platform] Add custom default max tokens

[gmarinho2]

FIX: vllm-project/vllm-spyre#148

Currently the default max tokens for the completions API is set to max_model_len - prompt_len. The changes in this PR make so that when a platform needs to use a different value for default_max_tokens it can be altered simply by overriding the maybe_update_max_tokens method in the class Plataform. When it is not needed it returns the current default. Edit: typo in commit message: class Plataform is meant to be class Platform.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[maxdebayser]

LGTM

[joerunde]

Is this something that can be handled by --generation-config?

--generation-config
The folder path to the generation config. Defaults to "auto", the generation config will be loaded from model path. If set to "vllm", no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If max_new_tokens is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests.

Default: 'auto'

Should the chat api be respecting a max_new_tokens override from the generation config instead of setting the default to max_model_len - prompt_len? That would allow a default override to be set regardless of platforms.

That said, I do like the code hook to be able to write whatever code you want too...

[maxdebayser]

Is this something that can be handled by --generation-config?
Unfortunately not because in vllm-spyre the permissible max_new_tokens depends on the request.

[NickLucche]

Thanks for the contribution!

Shouldn't we update serving_completion too?

[gmarinho2]

Thanks for the contribution!

Shouldn't we update serving_completion too?

Done. Since class CompletionRequest has 16 as default it will probably be selected most of the time because the default is set to be the minimum between context window, user request & server limit: REF1, REF2, REF3, REF4.

[joerunde]

@youkaichao any thoughts on getting this merged?

[NickLucche]

lgtm! Apologies for the delay.

[joerunde]

Let's get the full CI running then and see if we can get a maintainer to get this merged 👍

[joerunde]

@njhill can you hit the big green merge button for us?

[aarnphm]

One formatting comment, otherwise lgtm