MacOS Install with Metal GPU (original) (raw)

(1) Make sure you have xcode installed... at least the command line parts

[](#%5F%5Fcodelineno-0-1)# check the path of your xcode install [](#%5F%5Fcodelineno-0-2)xcode-select -p [](#%5F%5Fcodelineno-0-3) [](#%5F%5Fcodelineno-0-4)# xcode installed returns [](#%5F%5Fcodelineno-0-5)# /Applications/Xcode-beta.app/Contents/Developer [](#%5F%5Fcodelineno-0-6) [](#%5F%5Fcodelineno-0-7)# if xcode is missing then install it... it takes ages; [](#%5F%5Fcodelineno-0-8)xcode-select --install

(2) Install the conda version for MacOS that supports Metal GPU

[](#%5F%5Fcodelineno-1-1)wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh [](#%5F%5Fcodelineno-1-2)bash Miniforge3-MacOSX-arm64.sh

(3) Make a conda environment

[](#%5F%5Fcodelineno-2-1)conda create -n llama python=3.9.16 [](#%5F%5Fcodelineno-2-2)conda activate llama

(4) Install the LATEST llama-cpp-python...which happily supports MacOS Metal GPU as of version 0.1.62
(you needed xcode installed in order pip to build/compile the C++ code)

[](#%5F%5Fcodelineno-3-1)pip uninstall llama-cpp-python -y [](#%5F%5Fcodelineno-3-2)CMAKE_ARGS="-DGGML_METAL=on" pip install -U llama-cpp-python --no-cache-dir [](#%5F%5Fcodelineno-3-3)pip install 'llama-cpp-python[server]' [](#%5F%5Fcodelineno-3-4) [](#%5F%5Fcodelineno-3-5)# you should now have llama-cpp-python v0.1.62 or higher installed [](#%5F%5Fcodelineno-3-6)llama-cpp-python         0.1.68

(5) Download a v3 gguf v2 model - ggufv2 - file name ends with Q4_0.gguf - indicating it is 4bit quantized, with quantisation method 0

https://huggingface.co/TheBloke/CodeLlama-7B-GGUF

(6) run the llama-cpp-python API server with MacOS Metal GPU support

[](#%5F%5Fcodelineno-4-1)# config your ggml model path [](#%5F%5Fcodelineno-4-2)# make sure it is gguf v2 [](#%5F%5Fcodelineno-4-3)# make sure it is q4_0 [](#%5F%5Fcodelineno-4-4)export MODEL=[path to your llama.cpp ggml models]]/[ggml-model-name]]Q4_0.gguf [](#%5F%5Fcodelineno-4-5)python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1

Note: If you omit the --ngpulayers 1 then CPU will be used