GPU Device — OpenVINO™ documentation (original) (raw)

The GPU plugin is an OpenCL based plugin for inference of deep neural networks on Intel GPUs, both integrated and discrete ones. For an in-depth description of the GPU plugin, see:

The GPU plugin is a part of the Intel® Distribution of OpenVINO™ toolkit. For more information on how to configure a system to use it, see the GPU configuration.

Device Naming Convention#

For demonstration purposes, see the Hello Query Device C++ Sample that can print out the list of available devices with associated indices. Below is an example output (truncated to the device names only):

./hello_query_device Available devices: Device: CPU ... Device: GPU.0 ... Device: GPU.1

Then, the device name can be passed to the ov::Core::compile_model() method, running on:

default device

Python

core = ov.Core()
compiled_model = core.compile_model(model, "GPU")

C++

ov::Core core;
auto model = core.read_model("model.xml");
auto compiled_model = core.compile_model(model, "GPU");

specific GPU

Python

core = ov.Core()
compiled_model = core.compile_model(model, "GPU.1")

C++

ov::Core core;
auto model = core.read_model("model.xml");
auto compiled_model = core.compile_model(model, "GPU.1");

specific tile

Python

core = ov.Core()
compiled_model = core.compile_model(model, "GPU.1.0")

C++

ov::Core core;
auto model = core.read_model("model.xml");
auto compiled_model = core.compile_model(model, "GPU.1.0");

Supported Inference Data Types#

The GPU plugin supports the following data types as inference precision of internal primitives:

Selected precision of each primitive depends on the operation precision in IR, quantization primitives, and available hardware capabilities. The u1/u8/i8 data types are used for quantized operations only, which means that they are not selected automatically for non-quantized operations. For more details on how to get a quantized model, refer to the Model Optimization guide.

Floating-point precision of a GPU primitive is selected based on operation precision in the OpenVINO IR, except for the <compressed f16 OpenVINO IR form, which is executed in the f16 precision.

Note

The newer generation Intel Iris Xe and Xe MAX GPUs provide accelerated performance for i8/u8 models. Hardware acceleration for i8/u8 precision may be unavailable on older generation platforms. In such cases, a model is executed in the floating-point precision taken from IR. Hardware support of u8/i8 acceleration can be queried via the ov::device::capabilities property.

Hello Query Device C++ Sample can be used to print out the supported data types for all detected devices.

Supported Features#

The GPU plugin supports the following features:

Automatic Device Selection#

If a system has multiple GPUs (for example, an integrated and a discrete Intel GPU), then any supported model can be executed on all GPUs simultaneously. It is done by specifying AUTO:GPU.1,GPU.0 as a target device, and adding the CUMULATIVE_THROUGHPUT parameter.

Python

core = ov.Core()
compiled_model = core.compile_model(model, "AUTO:GPU.1,CPU.0", {hints.performance_mode: hints.PerformanceMode.CUMULATIVE_THROUGHPUT})

C++

ov::Core core;
auto model = core.read_model("model.xml");
auto compiled_model = core.compile_model(model, "AUTO:GPU.1,CPU.0", ov::hint::performance_mode(ov::hint::PerformanceMode::CUMULATIVE_THROUGHPUT));

For more details, see the Automatic Device Selection.

Automatic Batching#

The GPU plugin is capable of reporting ov::max_batch_size and ov::optimal_batch_size metrics with respect to the current hardware platform and model. Therefore, automatic batching is enabled by default when ov::optimal_batch_size is > 1 and ov::hint::performance_mode(ov::hint::PerformanceMode::THROUGHPUT) is set. Alternatively, it can be enabled explicitly via the device notion, for example BATCH:GPU.

Batching via BATCH plugin

Python

core = ov.Core()
compiled_model = core.compile_model(model, "BATCH:GPU")

C++

ov::Core core;
auto model = core.read_model("model.xml");
auto compiled_model = core.compile_model(model, "BATCH:GPU");

Batching via throughput hint

Python

import openvino.properties.hint as hints

core = ov.Core()
compiled_model = core.compile_model(
    model,
    "GPU",
    {
        hints.performance_mode: hints.PerformanceMode.THROUGHPUT,
    },
)

C++

ov::Core core;
auto model = core.read_model("model.xml");
auto compiled_model = core.compile_model(model, "GPU", ov::hint::performance_mode(ov::hint::PerformanceMode::THROUGHPUT));

For more details, see the Automatic batching.

Multi-stream Execution#

If either the ov::num_streams(n_streams) with n_streams > 1 or the ov::hint::performance_mode(ov::hint::PerformanceMode::THROUGHPUT) property is set for the GPU plugin, multiple streams are created for the model. In the case of GPU plugin each stream has its own host thread and an associated OpenCL queue which means that the incoming infer requests can be processed simultaneously.

Note

Simultaneous scheduling of kernels to different queues does not mean that the kernels are actually executed in parallel on the GPU device. The actual behavior depends on the hardware architecture and in some cases the execution may be serialized inside the GPU driver.

When multiple inferences of the same model need to be executed in parallel, the multi-stream feature is preferred to multiple instances of the model or application. The reason for this is that the implementation of streams in the GPU plugin supports weight memory sharing across streams, thus, memory consumption may be lower, compared to the other approaches.

For more details, see the optimization guide.

Dynamic Shapes#

Note

Currently, dynamic shape support for GPU is a preview feature and has the following limitations:

The general description of what dynamic shapes are and how they are used can be found indynamic shapes guide. To support dynamic shape execution, the following basic infrastructures are implemented:

Bounded dynamic batch#

It is worth noting that the internal behavior differs in the case of bounded-batch dynamic shapes, which means that only the batch dimension is dynamic and it has a fixed upper bound.

While general dynamic shapes can run on one compiled model, for the bounded dynamic batch the GPU plugin creates log2(N)low-level execution graphs in batch sizes equal to the powers of 2, to emulate the dynamic behavior (N - is the upper bound for the batch dimension here). As a result, the incoming infer request with a specific batch size is executed via the minimal combination of internal networks. For example, a batch size of 33 may be executed via two internal networks with batch sizes of 32 and 1. This approach is adopted for performance reasons, but it requires more memory and increased compilation time for multiple copies of internal networks.

The code snippet below demonstrates examples of a bounded dynamic batch:

Python

core = ov.Core()

C = 3
H = 224
W = 224

model.reshape([(1, 10), C, H, W])

# compile model and create infer request
compiled_model = core.compile_model(model, "GPU")
infer_request = compiled_model.create_infer_request()

# create input tensor with specific batch size
input_tensor = ov.Tensor(model.input().element_type, [2, C, H, W])

# ...

results = infer_request.infer([input_tensor])

C++

// Read model ov::Core core; auto model = core.read_model("model.xml");

model->reshape({{ov::Dimension(1, 10), ov::Dimension(C), ov::Dimension(H), ov::Dimension(W)}}); // {1..10, C, H, W}

// compile model and create infer request auto compiled_model = core.compile_model(model, "GPU"); auto infer_request = compiled_model.create_infer_request(); auto input = model->get_parameters().at(0);

// ...

// create input tensor with specific batch size ov::Tensor input_tensor(input->get_element_type(), {2, C, H, W});

// ...

infer_request.set_tensor(input, input_tensor); infer_request.infer();

Notes for performance and memory consumption in dynamic shapes#

Recommendations for performance improvement#

Preprocessing Acceleration#

The GPU plugin has the following additional preprocessing options:

Python

C++

using namespace ov::preprocess;
auto p = PrePostProcessor(model);
p.input().tensor().set_element_type(ov::element::u8)
                  .set_color_format(ov::preprocess::ColorFormat::NV12_TWO_PLANES, {"y", "uv"})
                  .set_memory_type(ov::intel_gpu::memory_type::surface);
p.input().preprocess().convert_color(ov::preprocess::ColorFormat::BGR);
p.input().model().set_layout("NCHW");
auto model_with_preproc = p.build();

With such preprocessing, GPU plugin will expect ov::intel_gpu::ocl::ClImage2DTensor (or derived) to be passed for each NV12 plane via ov::InferRequest::set_tensor() or ov::InferRequest::set_tensors() methods.

For usage examples, refer to the RemoteTensor API.

For more details, see the preprocessing API.

Model Caching#

Model Caching helps reduce application startup delays by exporting and reusing the compiled model automatically. The cache for the GPU plugin may be enabled via the common OpenVINO ov::cache_dir property.

This means that all plugin-specific model transformations are executed on each ov::Core::compile_model()call, regardless of the ov::cache_dir option. Still, since kernel compilation is a bottleneck in the model loading process, a significant load time reduction can be achieved. Currently, GPU plugin implementation fully supports static models only. For dynamic models, kernel caching is used instead and multiple ‘.cl_cache’ files are generated along with the ‘.blob’ file.

For more details, see the Model caching overview.

Extensibility#

For information on this subject, see the GPU Extensibility.

GPU Context and Memory Sharing via RemoteTensor API#

For information on this subject, see the RemoteTensor API of GPU Plugin.

Supported Properties#

The plugin supports the properties listed below.

Read-write properties#

All parameters must be set before calling ov::Core::compile_model() in order to take effect or passed as additional argument to ov::Core::compile_model().

Read-only Properties#

Limitations#

In some cases, the GPU plugin may implicitly execute several primitives on CPU using internal implementations, which may lead to an increase in CPU utilization. Below is a list of such operations:

The behavior depends on specific parameters of the operations and hardware configuration.

Important

While working on a fine tuned model, inference may give an inaccuracy and performance drop on GPU if winograd convolutions are selected. This issue can be fixed by disabling winograd convolutions:

compiled_model = core.compile_model(ov_model, device_name=devStr1, config={ "GPU_DISABLE_WINOGRAD_CONVOLUTION": True })

GPU Performance Checklist: Summary#

Since OpenVINO relies on the OpenCL kernels for the GPU implementation, many general OpenCL tips apply:

Additional Resources#