Running on AMD GPU (original) (raw)

September 2023, tested on 7900 XTX

Following the great instructions from August and using the docker image, this runs on the 7900 XTX with a few changes, most notably

export HSA_OVERRIDE_GFX_VERSION=11.0.0 #7900 xtx natively works with the gfx1100 driver make hip ROCM_TARGET=gfx1100

The rest of the steps are the same

August 2023, tested on 6900 XT and 6600 XT

Due to the great work of Odonata (Discord, github @edt-xx), the hardware of oceanmasterza (Discord), and the help of epicx (Discord, GitHub @bennmann), we have the below AMD instructions.

According the the author of the bitsandbytes ROCM port @arlo-phoenix, using a Docker image is recommended (both rocm/pytorch and rocm/pytorch-nightly should work). See port discussion here.

On host machine, run:

docker pull rocm/pytorch-nightly sudo docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-nightly

In the running image, run:

cd /home export HSA_OVERRIDE_GFX_VERSION=10.3.0

Install bitsandbytes with ROCM support

git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6.git bitsandbytes cd bitsandbytes make hip ROCM_TARGET=gfx1030 pip install pip --upgrade pip install .

Install Petals

cd .. pip install --upgrade git+https://github.com/bigscience-workshop/petals

Run server

python -m petals.cli.run_server petals-team/StableBeluga2 --port --torch_dtype float16

Running the model in bfloat16 is also supported but slower than in float16.

Multi-GPU process (--tensor_parallel_devices) is still not tested (docker --gpu flag may not function at this time and other virtualization tools may be necessary).

July 2023, tested on 6900 XT and 6600 XT

Contributed by: @edt-xx, @bennmann

Tested on:

• AMD 6600 XT tested July 24th, 2023 on Arch Linux with Rocm 5.6.0, mesa 22.1.4
• AMD 6900 XT tested April 18th, 2023 on bare metal Ubuntu 22.04 (no docker/anaconda/container). Tested with ROCM 5.4.2
• Untested on 7000 series, however 7000s may have much better performance as AMD added machine learning tensor library and better hardware support (vs ray tracing only on 6000 series)

Guide:

• use the mesa-clover and mesa-rusticl opencl variants
• add export HSA_OVERRIDE_GFX_VERSION=10.3.0 to your environment (put it to /home/user/.bashrc on ubuntu - this tricks ROCM to work on more consumer based cards like the 6000 series)
• install ROCM. Use this tutorial for Arch Linux: https://wiki.archlinux.org/title/GPGPU
• create and activate a venv for petals using python 3.11
  • python -m venv
  • cd
  • source bin/activate
• in the venv install pytorch, nightly version, with the command generated on by the website: https://pytorch.org/get-started/locally/
• install the Petals version with AMD GPU support:
  pip install git+https://github.com/bigscience-workshop/petals@amd-gpus

  This branch uses an older version of bitsandbytes patched to have AMD GPU support (developed by @brontoc and Titaniumtown). This means that you won't be able to use the 4-bit qunatization (--quant_type nf4) and LoRA adapters (the --adapters argument). The server will use 8-bit quantization (int8) for all models by default.
  Tip: You can set your fans to full speed or close to it before starting Petals (the default Linux fan profile for AMD GPUs is not good on some cards): rocm-smi --setfan 99%

• run petals using:
  python -m petals.cli.run_server petals-team/StableBeluga2
  Tip: You can monitor temperature and woltage by running this: rocm-smi && rocm-smi -t