LiteRT Core ML delegate (original) (raw)

The LiteRT Core ML delegate enables running LiteRT models onCore ML framework, which results in faster model inference on iOS devices.

Supported iOS versions and devices:

Supported models

The Core ML delegate currently supports float (FP32 and FP16) models.

Trying the Core ML delegate on your own model

The Core ML delegate is already included in nightly release of LiteRT CocoaPods. To use Core ML delegate, change your LiteRT pod to include subspec CoreML in your Podfile.

target 'YourProjectName'
  pod 'TensorFlowLiteSwift/CoreML', '~> 2.4.0'  # Or TensorFlowLiteObjC/CoreML

OR

# Particularily useful when you also want to include 'Metal' subspec.
target 'YourProjectName'
  pod 'TensorFlowLiteSwift', '~> 2.4.0', :subspecs => ['CoreML']

Swift

let coreMLDelegate = CoreMLDelegate()
var interpreter: Interpreter

// Core ML delegate will only be created for devices with Neural Engine
if coreMLDelegate != nil {
  interpreter = try Interpreter(modelPath: modelPath,
                                delegates: [coreMLDelegate!])
} else {
  interpreter = try Interpreter(modelPath: modelPath)
}

Objective-C

// Import module when using CocoaPods with module support
@import TFLTensorFlowLite;

// Or import following headers manually
# import "tensorflow/lite/objc/apis/TFLCoreMLDelegate.h"
# import "tensorflow/lite/objc/apis/TFLTensorFlowLite.h"

// Initialize Core ML delegate
TFLCoreMLDelegate* coreMLDelegate = [[TFLCoreMLDelegate alloc] init];

// Initialize interpreter with model path and Core ML delegate
TFLInterpreterOptions* options = [[TFLInterpreterOptions alloc] init];
NSError* error = nil;
TFLInterpreter* interpreter = [[TFLInterpreter alloc]
                                initWithModelPath:modelPath
                                          options:options
                                        delegates:@[ coreMLDelegate ]
                                            error:&error];
if (error != nil) { /* Error handling... */ }

if (![interpreter allocateTensorsWithError:&error]) { /* Error handling... */ }
if (error != nil) { /* Error handling... */ }

// Run inference ...

C (Until 2.3.0)

#include "tensorflow/lite/delegates/coreml/coreml_delegate.h"

// Initialize interpreter with model
TfLiteModel* model = TfLiteModelCreateFromFile(model_path);

// Initialize interpreter with Core ML delegate
TfLiteInterpreterOptions* options = TfLiteInterpreterOptionsCreate();
TfLiteDelegate* delegate = TfLiteCoreMlDelegateCreate(NULL);  // default config
TfLiteInterpreterOptionsAddDelegate(options, delegate);
TfLiteInterpreterOptionsDelete(options);

TfLiteInterpreter* interpreter = TfLiteInterpreterCreate(model, options);

TfLiteInterpreterAllocateTensors(interpreter);

// Run inference ...

/* ... */

// Dispose resources when it is no longer used.
// Add following code to the section where you dispose of the delegate
// (e.g. `dealloc` of class).

TfLiteInterpreterDelete(interpreter);
TfLiteCoreMlDelegateDelete(delegate);
TfLiteModelDelete(model);

Best practices

Using Core ML delegate on devices without Neural Engine

By default, Core ML delegate will only be created if the device has Neural Engine, and will return null if the delegate is not created. If you want to run Core ML delegate on other environments (for example, simulator), pass .allas an option while creating delegate in Swift. On C++ (and Objective-C), you can pass TfLiteCoreMlDelegateAllDevices. Following example shows how to do this:

Swift

var options = CoreMLDelegate.Options()
options.enabledDevices = .all
let coreMLDelegate = CoreMLDelegate(options: options)!
let interpreter = try Interpreter(modelPath: modelPath,
                                  delegates: [coreMLDelegate])

Objective-C

TFLCoreMLDelegateOptions* coreMLOptions = [[TFLCoreMLDelegateOptions alloc] init];
coreMLOptions.enabledDevices = TFLCoreMLDelegateEnabledDevicesAll;
TFLCoreMLDelegate* coreMLDelegate = [[TFLCoreMLDelegate alloc]
                                      initWithOptions:coreMLOptions];

// Initialize interpreter with delegate

C

TfLiteCoreMlDelegateOptions options;
options.enabled_devices = TfLiteCoreMlDelegateAllDevices;
TfLiteDelegate* delegate = TfLiteCoreMlDelegateCreate(&options);
// Initialize interpreter with delegate

Using Metal(GPU) delegate as a fallback.

When the Core ML delegate is not created, alternatively you can still useMetal delegate to get performance benefits. Following example shows how to do this:

Swift

var delegate = CoreMLDelegate()
if delegate == nil {
  delegate = MetalDelegate()  // Add Metal delegate options if necessary.
}

let interpreter = try Interpreter(modelPath: modelPath,
                                  delegates: [delegate!])

Objective-C

TFLDelegate* delegate = [[TFLCoreMLDelegate alloc] init];
if (!delegate) {
  // Add Metal delegate options if necessary
  delegate = [[TFLMetalDelegate alloc] init];
}
// Initialize interpreter with delegate

C

TfLiteCoreMlDelegateOptions options = {};
delegate = TfLiteCoreMlDelegateCreate(&options);
if (delegate == NULL) {
  // Add Metal delegate options if necessary
  delegate = TFLGpuDelegateCreate(NULL);
}
// Initialize interpreter with delegate

The delegate creation logic reads device's machine id (e.g. iPhone11,1) to determine its Neural Engine availability. See thecodefor more detail. Alternatively, you can implement your own set of denylist devices using other libraries such asDeviceKit.

Using older Core ML version

Although iOS 13 supports Core ML 3, the model might work better when it is converted with Core ML 2 model specification. The target conversion version is set to the latest version by default, but you can change this by settingcoreMLVersion (in Swift, coreml_version in C API) in the delegate option to older version.

Supported ops

Following ops are supported by the Core ML delegate.

Feedback

For issues, please create aGitHubissue with all the necessary details to reproduce.

FAQ

APIs