Installing on Grace Hopper — TensorRT-LLM (original) (raw)

1. Install TensorRT-LLM (tested on Ubuntu 24.04).
   pip3 install torch==2.6.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
   sudo apt-get -y install libopenmpi-dev && pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm
   If using the PyTorch NGC Container image, the prerequisite step for installing CUDA-enabled PyTorch package is not required.
2. Sanity check the installation by running the following in Python (tested on Python 3.12):
   1from tensorrt_llm import LLM, SamplingParams
   2
   3
   4def main():
   5
   6 prompts = [
   7 "Hello, my name is",

8 "The president of the United States is",
9 "The capital of France is",
10 "The future of AI is",
11 ]
12 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
13
14 llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
15
16 outputs = llm.generate(prompts, sampling_params)
17
18 # Print the outputs.
19 for output in outputs:
20 prompt = output.prompt
21 generated_text = output.outputs[0].text
22 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
23
24
25# The entry point of the program need to be protected for spawning processes.
26if name == 'main':
27 main()