[Bugfix] Fix cpu-offload-gb assertion with non-default block sizes by AjAnubolu · Pull Request #36461 · vllm-project/vllm (original) (raw)
Signed-off-by: AjAnubolu anuboluajay@gmail.com
[![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4)
Signed-off-by: Michael Goin mgoin64@gmail.com
mtparet pushed a commit to blackfuel-ai/vllm that referenced this pull request
Signed-off-by: AjAnubolu anuboluajay@gmail.com Signed-off-by: Michael Goin mgoin64@gmail.com Co-authored-by: Michael Goin mgoin64@gmail.com
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request
Signed-off-by: AjAnubolu anuboluajay@gmail.com Signed-off-by: Michael Goin mgoin64@gmail.com Co-authored-by: Michael Goin mgoin64@gmail.com
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request
Signed-off-by: AjAnubolu anuboluajay@gmail.com Signed-off-by: Michael Goin mgoin64@gmail.com Co-authored-by: Michael Goin mgoin64@gmail.com
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request
Signed-off-by: AjAnubolu anuboluajay@gmail.com Signed-off-by: Michael Goin mgoin64@gmail.com Co-authored-by: Michael Goin mgoin64@gmail.com
jhu960213 pushed a commit to jhu960213/vllm that referenced this pull request
Signed-off-by: AjAnubolu anuboluajay@gmail.com Signed-off-by: Michael Goin mgoin64@gmail.com Co-authored-by: Michael Goin mgoin64@gmail.com
mvanhorn pushed a commit to mvanhorn/vllm that referenced this pull request
Signed-off-by: AjAnubolu anuboluajay@gmail.com Signed-off-by: Michael Goin mgoin64@gmail.com Co-authored-by: Michael Goin mgoin64@gmail.com Signed-off-by: Matt Van Horn 455140+mvanhorn@users.noreply.github.com
wenyili added a commit to wenyili/vllm that referenced this pull request
Move InputBatch creation from GPUModelRunner.init to initialize_kv_cache (via a new initialize_input_batch method), so it is built with the final block sizes from kv_cache_config rather than a placeholder.
The original early initialization was a workaround for a UVA pinned-memory reuse bug (see vllm-project#18298): GPTQ's process_weights_after_loading replaced parameter objects, causing the old PackedvLLMParameter (which held the only Python reference to cpu_data) to be GC'd and its pinned memory returned to CachingHostAllocator. InputBatch, if created after load_model, would then reuse that memory for block_table_cpu, aliasing live GPTQ weight CUDA views.
This is now safe because the C++ lambda in csrc/cuda_view.cu captures cpu_tensor by value (base = cpu_tensor{}), keeping it alive for the lifetime of the UVA CUDA view regardless of Python-side GC. PR vllm-project#36461 confirmed this by removing the offload+quantization reinit guard added in vllm-project#18654.
The may_reinitialize_input_batch method is renamed to initialize_input_batch and the conditional block-size comparison is dropped — InputBatch is always created fresh in initialize_kv_cache. This also fixes a latent bug where cp_kv_cache_interleave_size was omitted from the reinit path.
Co-authored-by: Claude Signed-off-by: liwenyi liwenyi199111@gmail.com
Signed-off-by: liwenyi lwy.lwy@163.com
wenyili added a commit to wenyili/vllm that referenced this pull request
Move InputBatch creation from GPUModelRunner.init to initialize_kv_cache (via a new initialize_input_batch method), so it is built with the final block sizes from kv_cache_config rather than a placeholder.
The original early initialization was a workaround for a UVA pinned-memory reuse bug (see vllm-project#18298): GPTQ's process_weights_after_loading replaced parameter objects, causing the old PackedvLLMParameter (which held the only Python reference to cpu_data) to be GC'd and its pinned memory returned to CachingHostAllocator. InputBatch, if created after load_model, would then reuse that memory for block_table_cpu, aliasing live GPTQ weight CUDA views.
This is now safe because the C++ lambda in csrc/cuda_view.cu captures cpu_tensor by value (base = cpu_tensor{}), keeping it alive for the lifetime of the UVA CUDA view regardless of Python-side GC. PR vllm-project#36461 confirmed this by removing the offload+quantization reinit guard added in vllm-project#18654.
The may_reinitialize_input_batch method is renamed to initialize_input_batch and the conditional block-size comparison is dropped — InputBatch is always created fresh in initialize_kv_cache. This also fixes a latent bug where cp_kv_cache_interleave_size was omitted from the reinit path.
Co-authored-by: Claude Signed-off-by: liwenyi liwenyi199111@gmail.com
Signed-off-by: liwenyi lwy.lwy@163.com
wenyili added a commit to wenyili/vllm that referenced this pull request
Move InputBatch creation from GPUModelRunner.init to initialize_kv_cache (via a new initialize_input_batch method), so it is built with the final block sizes from kv_cache_config rather than a placeholder.
The original early initialization was a workaround for a UVA pinned-memory reuse bug (see vllm-project#18298): GPTQ's process_weights_after_loading replaced parameter objects, causing the old PackedvLLMParameter (which held the only Python reference to cpu_data) to be GC'd and its pinned memory returned to CachingHostAllocator. InputBatch, if created after load_model, would then reuse that memory for block_table_cpu, aliasing live GPTQ weight CUDA views.
This is now safe because the C++ lambda in csrc/cuda_view.cu captures cpu_tensor by value (base = cpu_tensor{}), keeping it alive for the lifetime of the UVA CUDA view regardless of Python-side GC. PR vllm-project#36461 confirmed this by removing the offload+quantization reinit guard added in vllm-project#18654.
The may_reinitialize_input_batch method is renamed to initialize_input_batch and the conditional block-size comparison is dropped — InputBatch is always created fresh in initialize_kv_cache. This also fixes a latent bug where cp_kv_cache_interleave_size was omitted from the reinit path.
Co-authored-by: Claude Signed-off-by: liwenyi lwy.lwy@163.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})