[Bugfix] Support eos_token_id from config.json by DarkLight1337 · Pull Request #5954 · vllm-project/vllm (original) (raw)

While working on BLIP-2 in #5920, I found that the model kept repeating the output in a fashion similar to Llama 3 when it was first released. It turns out that this is because vLLM failed to consider the extra EOS token (\n) which is only specified in config.json rather than generation_config.json or tokenizer_config.json.

Building on #4182, this PR adds another fallback when loading EOS tokens from the generation config. I have tested this on a local copy of #5920 and it successfully resolves my issue.