Throughput Benchmark Sample — OpenVINO™ documentation (original) (raw)

This sample demonstrates how to estimate performance of a model using Asynchronous Inference Request API in throughput mode. This sample does not have other configurable command-line arguments. Feel free to modify sample’s source code to try out different options.

The reported results may deviate from what benchmark_appreports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.

Before using the sample, refer to the following requirements:

The sample accepts any file format supported by core.read_model.
The sample has been validated with: yolo-v3-tf and face-detection-0200 models.
To build the sample, use instructions available at Build the Sample Applicationssection in “Get Started with Samples” guide.

How It Works#

The sample compiles a model for a given device, randomly generates input data, performs asynchronous inference multiple times for a given number of seconds. Then, it processes and reports performance results.

You can see the explicit description of each sample step atIntegration Stepssection of “Integrate OpenVINO™ Runtime with Your Application” guide.

Running#

Python

python throughput_benchmark.py (default: CPU)

C++

throughput_benchmark (default: CPU)

To run the sample, you need to specify a model. You can get a model specific for your inference task from one of model repositories, such as TensorFlow Zoo, HuggingFace, or TensorFlow Hub.

Example#

Download a pre-trained model.
You can convert it by using:
Python
import openvino as ov
ov_model = ov.convert_model('./models/googlenet-v1')

or, when model is a Python model object

ov_model = ov.convert_model(googlenet-v1)
CLI
ovc ./models/googlenet-v1 3. Perform benchmarking, using the googlenet-v1 model on a CPU:
Python
python throughput_benchmark.py ./models/googlenet-v1.xml
C++
throughput_benchmark ./models/googlenet-v1.xml

Sample Output#

Python

The application outputs performance results.

[ INFO ] OpenVINO: [ INFO ] Build ................................. [ INFO ] Count: 2817 iterations [ INFO ] Duration: 10012.65 ms [ INFO ] Latency: [ INFO ] Median: 13.80 ms [ INFO ] Average: 14.10 ms [ INFO ] Min: 8.35 ms [ INFO ] Max: 28.38 ms [ INFO ] Throughput: 281.34 FPS

C++