[Don't merge] Debug failing quantization test with input batch move by heheda12345 · Pull Request #18298 · vllm-project/vllm (original) (raw)
Signed-off-by: Chen Zhang zhangch99@outlook.com
jomcgi added a commit to jomcgi/homelab that referenced this pull request
torch.compile in vLLM v0.19.1 adds ~5 GiB overhead, pushing total VRAM past the 4090's 23.5 GiB. --enforce-eager disables compilation, keeping model at ~17.5 GiB with headroom for KV cache. cpu-offload-gb is broken in v0.19.1 (vllm-project/vllm#18298).
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
jomcgi added a commit to jomcgi/homelab that referenced this pull request
torch.compile in vLLM v0.19.1 adds ~5 GiB overhead, pushing total VRAM past the 4090's 23.5 GiB. --enforce-eager disables compilation, keeping model at ~17.5 GiB with headroom for KV cache. cpu-offload-gb is broken in v0.19.1 (vllm-project/vllm#18298).
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
wenyili added a commit to wenyili/vllm that referenced this pull request
Move InputBatch creation from GPUModelRunner.init to initialize_kv_cache (via a new initialize_input_batch method), so it is built with the final block sizes from kv_cache_config rather than a placeholder.
The original early initialization was a workaround for a UVA pinned-memory reuse bug (see vllm-project#18298): GPTQ's process_weights_after_loading replaced parameter objects, causing the old PackedvLLMParameter (which held the only Python reference to cpu_data) to be GC'd and its pinned memory returned to CachingHostAllocator. InputBatch, if created after load_model, would then reuse that memory for block_table_cpu, aliasing live GPTQ weight CUDA views.
This is now safe because the C++ lambda in csrc/cuda_view.cu captures cpu_tensor by value (base = cpu_tensor{}), keeping it alive for the lifetime of the UVA CUDA view regardless of Python-side GC. PR vllm-project#36461 confirmed this by removing the offload+quantization reinit guard added in vllm-project#18654.
The may_reinitialize_input_batch method is renamed to initialize_input_batch and the conditional block-size comparison is dropped — InputBatch is always created fresh in initialize_kv_cache. This also fixes a latent bug where cp_kv_cache_interleave_size was omitted from the reinit path.
Co-authored-by: Claude Signed-off-by: liwenyi liwenyi199111@gmail.com
Signed-off-by: liwenyi lwy.lwy@163.com
wenyili added a commit to wenyili/vllm that referenced this pull request
Move InputBatch creation from GPUModelRunner.init to initialize_kv_cache (via a new initialize_input_batch method), so it is built with the final block sizes from kv_cache_config rather than a placeholder.
The original early initialization was a workaround for a UVA pinned-memory reuse bug (see vllm-project#18298): GPTQ's process_weights_after_loading replaced parameter objects, causing the old PackedvLLMParameter (which held the only Python reference to cpu_data) to be GC'd and its pinned memory returned to CachingHostAllocator. InputBatch, if created after load_model, would then reuse that memory for block_table_cpu, aliasing live GPTQ weight CUDA views.
This is now safe because the C++ lambda in csrc/cuda_view.cu captures cpu_tensor by value (base = cpu_tensor{}), keeping it alive for the lifetime of the UVA CUDA view regardless of Python-side GC. PR vllm-project#36461 confirmed this by removing the offload+quantization reinit guard added in vllm-project#18654.
The may_reinitialize_input_batch method is renamed to initialize_input_batch and the conditional block-size comparison is dropped — InputBatch is always created fresh in initialize_kv_cache. This also fixes a latent bug where cp_kv_cache_interleave_size was omitted from the reinit path.
Co-authored-by: Claude Signed-off-by: liwenyi liwenyi199111@gmail.com
Signed-off-by: liwenyi lwy.lwy@163.com
wenyili added a commit to wenyili/vllm that referenced this pull request
Move InputBatch creation from GPUModelRunner.init to initialize_kv_cache (via a new initialize_input_batch method), so it is built with the final block sizes from kv_cache_config rather than a placeholder.
The original early initialization was a workaround for a UVA pinned-memory reuse bug (see vllm-project#18298): GPTQ's process_weights_after_loading replaced parameter objects, causing the old PackedvLLMParameter (which held the only Python reference to cpu_data) to be GC'd and its pinned memory returned to CachingHostAllocator. InputBatch, if created after load_model, would then reuse that memory for block_table_cpu, aliasing live GPTQ weight CUDA views.
This is now safe because the C++ lambda in csrc/cuda_view.cu captures cpu_tensor by value (base = cpu_tensor{}), keeping it alive for the lifetime of the UVA CUDA view regardless of Python-side GC. PR vllm-project#36461 confirmed this by removing the offload+quantization reinit guard added in vllm-project#18654.
The may_reinitialize_input_batch method is renamed to initialize_input_batch and the conditional block-size comparison is dropped — InputBatch is always created fresh in initialize_kv_cache. This also fixes a latent bug where cp_kv_cache_interleave_size was omitted from the reinit path.
Co-authored-by: Claude Signed-off-by: liwenyi lwy.lwy@163.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})