Adding load_in_8bit to inference_hf.py by airaria · Pull Request #756 · ymcui/Chinese-LLaMA-Alpaca (original) (raw)

Description

Adding --load_in_8bit option to inference_hf.py

Usage
Just add the --load_in_8bit to the launching command. For example:

python scripts/inference/inference_hf.py \
    --base_model path_to_merged_chinese_alpaca_plus \
    --with_prompt --interactive \
    --load_in_8bit

#747