NxD Inference Setup Guide — AWS Neuron Documentation (original) (raw)

Contents

• Option 1: Launch an instance using a Neuron DLAMI
• Option 2: Use a Neuron Deep Learning Container (DLC)
• Option 3: Manually Install NxD Inference
  • Setup a Neuron Environment
  • Install NxD Inference
• Verify NxD Inference Installation

NxD Inference Setup Guide#

The NeuronX Distributed (NxD) Inference framework is built on top ofNxD Core. Follow the steps in this guide to set up your environment to run inference using the NxD Inference framework.

Table of contents

• Option 1: Launch an instance using a Neuron DLAMI
• Option 2: Use a Neuron Deep Learning Container (DLC)
• Option 3: Manually Install NxD Inference
  • Setup a Neuron Environment
  • Install NxD Inference
• Verify NxD Inference Installation

Option 1: Launch an instance using a Neuron DLAMI#

Neuron Deep Learning AMIs (DLAMIs) are Amazon Machine Images (AMIs) that come with the Neuron SDK pre-installed. To quickly get started with NxD Inference, you can launch an EC2 instance with the multi-framework DLAMI, which includes NxD Inference and its dependencies. For more information, see theNeuron Multi-Framework DLAMI Guideand Neuron DLAMI User Guide.

After you launch an instance, you can run the following command to activate the NxD Inference virtual environment.

source /opt/aws_neuronx_venv_pytorch_2_6_nxd_inference/bin/activate

Option 2: Use a Neuron Deep Learning Container (DLC)#

Neuron Deep Learning Containers (DLCs) are Docker images that come with the Neuron SDK pre-installed. To run NxD Inference in a Docker container, use theNeuronx PyTorch Inference Containers. For more information, see Neuron Containers.

Option 3: Manually Install NxD Inference#

Follow these instructions to manually install NxD Inference on an instance.

Note

For information about which Python versions are compatible with the Neuron SDK, see Release Artifacts.

Setup a Neuron Environment#

Before you install NxD Inference, you must install the Neuron SDK and its dependencies, including PyTorch Neuron (torch-neuronx). Follow instructions for one of the following operating systems:

• PyTorch NeuronX Setup on Ubuntu 22
• PyTorch NeuronX Setup on Amazon Linux 2023

Install NxD Inference#

Run this command to install NxD Inference.

source aws_neuron_venv_pytorch/bin/activate pip install -U pip pip install --upgrade neuronx-cc==2.* neuronx-distributed-inference --extra-index-url https://pip.repos.neuron.amazonaws.com

Verify NxD Inference Installation#

To verify that NxD Inference installed successfully, check that you can run the inference_demo console script.