(original) (raw)

{ "cells": [ { "cell_type": "markdown", "id": "g_nWetWWd_ns", "metadata": { "id": "g_nWetWWd_ns" }, "source": [ "Project: /edge/_project.yaml\n", "Book: /edge/_book.yaml\n", "\n", "##### Copyright 2024 The AI Edge Authors." ] }, { "cell_type": "code", "execution_count": null, "id": "2pHVBk_seED1", "metadata": { "cellView": "form", "id": "2pHVBk_seED1" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "id": "M7vSdG6sAIQn", "metadata": { "id": "M7vSdG6sAIQn" }, "source": [ "# Signatures in LiteRT" ] }, { "cell_type": "markdown", "id": "fwc5GKHBASdc", "metadata": { "id": "fwc5GKHBASdc" }, "source": [ "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n", " \u003ctd\u003e\n", " \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/signatures.ipynb\\"\\u003e\\u003cimg src=\"https://www.tensorflow.org/images/colab\_logo\_32px.png\\" /\u003eRun in Google Colab\u003c/a\u003e\n", " \u003c/td\u003e\n", " \u003ctd\u003e\n", " \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/signatures.ipynb\\"\\u003e\\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\\" /\u003eView source on GitHub\u003c/a\u003e\n", " \u003c/td\u003e\n", " \u003ctd\u003e\n", " \u003ca href=\"https://storage.googleapis.com/tensorflow\_docs/tensorflow/tensorflow/lite/g3doc/guide/signatures.ipynb\\"\\u003e\\u003cimg src=\"https://www.tensorflow.org/images/download\_logo\_32px.png\\" /\u003eDownload notebook\u003c/a\u003e\n", " \u003c/td\u003e\n", "\u003c/table\u003e" ] }, { "cell_type": "markdown", "id": "9ee074e4", "metadata": { "id": "9ee074e4" }, "source": [ "LiteRT supports converting TensorFlow model's input/output\n", "specifications to LiteRT models. The input/output specifications are\n", "called \"signatures\". Signatures can be specified when building a SavedModel or\n", "creating concrete functions.\n", "\n", "Signatures in LiteRT provide the following features:\n", "\n", "* They specify inputs and outputs of the converted LiteRT model by\n", " respecting the TensorFlow model's signatures.\n", "* Allow a single LiteRT model to support multiple entry points.\n", "\n", "The signature is composed of three pieces:\n", "\n", "* Inputs: Map for inputs from input name in the signature to an input tensor.\n", "* Outputs: Map for output mapping from output name in signature to an output\n", " tensor.\n", "* Signature Key: Name that identifies an entry point of the graph.\n", "\n" ] }, { "cell_type": "markdown", "id": "UaWdLA3fQDK2", "metadata": { "id": "UaWdLA3fQDK2" }, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "9j4MGqyKQEo4", "metadata": { "id": "9j4MGqyKQEo4" }, "outputs": [], "source": [ "import tensorflow as tf" ] }, { "cell_type": "markdown", "id": "FN2N6hPEP-Ay", "metadata": { "id": "FN2N6hPEP-Ay" }, "source": [ "## Example model\n", "\n", "Let's say we have two tasks, e.g., encoding and decoding, as a TensorFlow model:" ] }, { "cell_type": "code", "execution_count": null, "id": "d8577c80", "metadata": { "id": "d8577c80" }, "outputs": [], "source": [ "class Model(tf.Module):\n", "\n", " @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.float32)])\n", " def encode(self, x):\n", " result = tf.strings.as_string(x)\n", " return {\n", " \"encoded_result\": result\n", " }\n", "\n", " @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.string)])\n", " def decode(self, x):\n", " result = tf.strings.to_number(x)\n", " return {\n", " \"decoded_result\": result\n", " }" ] }, { "cell_type": "markdown", "id": "9c814c6e", "metadata": { "id": "9c814c6e" }, "source": [ "In the signature wise, the above TensorFlow model can be summarized as follows:\n", "\n", "* Signature\n", "\n", " - Key: encode\n", " - Inputs: {\"x\"}\n", " - Output: {\"encoded_result\"}\n", "\n", "* Signature\n", "\n", " - Key: decode\n", " - Inputs: {\"x\"}\n", " - Output: {\"decoded_result\"}" ] }, { "cell_type": "markdown", "id": "c4099f20", "metadata": { "id": "c4099f20" }, "source": [ "## Convert a model with Signatures\n", "\n", "LiteRT converter APIs will bring the above signature information into\n", "the converted LiteRT model.\n", "\n", "This conversion functionality is available on all the converter APIs starting\n", "from TensorFlow version 2.7.0. See example usages.\n" ] }, { "cell_type": "markdown", "id": "Qv0WwFQkQgnO", "metadata": { "id": "Qv0WwFQkQgnO" }, "source": [ "\n", "### From Saved Model" ] }, { "cell_type": "code", "execution_count": null, "id": "96c8fc79", "metadata": { "id": "96c8fc79" }, "outputs": [], "source": [ "model = Model()\n", "\n", "# Save the model\n", "SAVED_MODEL_PATH = 'content/saved_models/coding'\n", "\n", "tf.saved_model.save(\n", " model, SAVED_MODEL_PATH,\n", " signatures={\n", " 'encode': model.encode.get_concrete_function(),\n", " 'decode': model.decode.get_concrete_function()\n", " })\n", "\n", "# Convert the saved model using TFLiteConverter\n", "converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL_PATH)\n", "converter.target_spec.supported_ops = [\n", " tf.lite.OpsSet.TFLITE_BUILTINS, # enable LiteRT ops.\n", " tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n", "]\n", "tflite_model = converter.convert()\n", "\n", "# Print the signatures from the converted model\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "signatures = interpreter.get_signature_list()\n", "print(signatures)" ] }, { "cell_type": "markdown", "id": "5baa9f17", "metadata": { "id": "5baa9f17" }, "source": [ "### From Keras Model" ] }, { "cell_type": "code", "execution_count": null, "id": "71f29229", "metadata": { "id": "71f29229" }, "outputs": [], "source": [ "# Generate a Keras model.\n", "keras_model = tf.keras.Sequential(\n", " [\n", " tf.keras.layers.Dense(2, input_dim=4, activation='relu', name='x'),\n", " tf.keras.layers.Dense(1, activation='relu', name='output'),\n", " ]\n", ")\n", "\n", "# Convert the keras model using TFLiteConverter.\n", "# Keras model converter API uses the default signature automatically.\n", "converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)\n", "tflite_model = converter.convert()\n", "\n", "# Print the signatures from the converted model\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "\n", "signatures = interpreter.get_signature_list()\n", "print(signatures)" ] }, { "cell_type": "markdown", "id": "e4d30f85", "metadata": { "id": "e4d30f85" }, "source": [ "### From Concrete Functions" ] }, { "cell_type": "code", "execution_count": null, "id": "c9e8a742", "metadata": { "id": "c9e8a742" }, "outputs": [], "source": [ "model = Model()\n", "\n", "# Convert the concrete functions using TFLiteConverter\n", "converter = tf.lite.TFLiteConverter.from_concrete_functions(\n", " [model.encode.get_concrete_function(),\n", " model.decode.get_concrete_function()], model)\n", "converter.target_spec.supported_ops = [\n", " tf.lite.OpsSet.TFLITE_BUILTINS, # enable LiteRT ops.\n", " tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n", "]\n", "tflite_model = converter.convert()\n", "\n", "# Print the signatures from the converted model\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "signatures = interpreter.get_signature_list()\n", "print(signatures)" ] }, { "cell_type": "markdown", "id": "b5e85934", "metadata": { "id": "b5e85934" }, "source": [ "## Run Signatures\n", "\n", "TensorFlow inference APIs support the signature-based executions:\n", "\n", "* Accessing the input/output tensors through the names of the inputs and\n", " outputs, specified by the signature.\n", "* Running each entry point of the graph separately, identified by the\n", " signature key.\n", "* Support for the SavedModel's initialization procedure.\n", "\n", "Java, C++ and Python language bindings are currently available. See example the\n", "below sections.\n" ] }, { "cell_type": "markdown", "id": "ZRBMFciMQmiB", "metadata": { "id": "ZRBMFciMQmiB" }, "source": [ "\n", "### Java" ] }, { "cell_type": "markdown", "id": "04c5a4fc", "metadata": { "id": "04c5a4fc" }, "source": [ "```\n", "try (Interpreter interpreter = new Interpreter(file_of_tensorflowlite_model)) {\n", " // Run encoding signature.\n", " Map\u003cString, Object\u003e inputs = new HashMap\u003c\u003e();\n", " inputs.put(\"x\", input);\n", " Map\u003cString, Object\u003e outputs = new HashMap\u003c\u003e();\n", " outputs.put(\"encoded_result\", encoded_result);\n", " interpreter.runSignature(inputs, outputs, \"encode\");\n", "\n", " // Run decoding signature.\n", " Map\u003cString, Object\u003e inputs = new HashMap\u003c\u003e();\n", " inputs.put(\"x\", encoded_result);\n", " Map\u003cString, Object\u003e outputs = new HashMap\u003c\u003e();\n", " outputs.put(\"decoded_result\", decoded_result);\n", " interpreter.runSignature(inputs, outputs, \"decode\");\n", "}\n", "```" ] }, { "cell_type": "markdown", "id": "5ba86c64", "metadata": { "id": "5ba86c64" }, "source": [ "### C++" ] }, { "metadata": { "id": "2BFOKaNBPdwR" }, "cell_type": "markdown", "source": [ "```\n", "SignatureRunner* encode_runner =\n", " interpreter-\u003eGetSignatureRunner(\"encode\");\n", "encode_runner-\u003eResizeInputTensor(\"x\", {100});\n", "encode_runner-\u003eAllocateTensors();\n", "\n", "TfLiteTensor* input_tensor = encode_runner-\u003einput_tensor(\"x\");\n", "float* input = GetTensorData\u003cfloat\u003e(input_tensor);\n", "// Fill `input`.\n", "\n", "encode_runner-\u003eInvoke();\n", "\n", "const TfLiteTensor* output_tensor = encode_runner-\u003eoutput_tensor(\n", " \"encoded_result\");\n", "float* output = GetTensorData\u003cfloat\u003e(output_tensor);\n", "// Access `output`.\n", "```" ] }, { "metadata": { "id": "Wjodm0IPPdwR" }, "cell_type": "markdown", "source": [ "### C" ] }, { "cell_type": "markdown", "id": "397ad6fd", "metadata": { "id": "397ad6fd" }, "source": [ "```\n", "TfLiteSignatureRunner* encode_runner =\n", " TfLiteInterpreterGetSignatureRunner(interpreter, \"encode\");\n", "const int dims[] = {100};\n", "const int num_dims = 1;\n", "TfLiteSignatureRunnerResizeInputTensor(encode_runner, \"x\", dims, num_dims);\n", "TfLiteSignatureRunnerAllocateTensors-\u003eAllocateTensors();\n", "\n", "TfLiteTensor* input_tensor =\n", " TfLiteSignatureRunnerGetInputTensor(encode_runner, \"x\");\n", "float* input = (float *)TFLiteTensorData(input_tensor);\n", "// Fill `input`.\n", "\n", "TfLiteSignatureRunnerInvoke(encode_runner);\n", "\n", "const TfLiteTensor* output_tensor = TfLiteSignatureRunnerGetOutputTensor(\n", " encode_runner, \"encoded_result\");\n", "const float* output = (const float *)TfLiteTensorData(output_tensor);\n", "// Access `output`.\n", "```" ] }, { "cell_type": "markdown", "id": "0f4c6ad4", "metadata": { "id": "0f4c6ad4" }, "source": [ "### Python" ] }, { "cell_type": "code", "execution_count": null, "id": "ab7b1963", "metadata": { "id": "ab7b1963" }, "outputs": [], "source": [ "# Load the LiteRT model in LiteRT Interpreter\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "\n", "# Print the signatures from the converted model\n", "signatures = interpreter.get_signature_list()\n", "print('Signature:', signatures)\n", "\n", "# encode and decode are callable with input as arguments.\n", "encode = interpreter.get_signature_runner('encode')\n", "decode = interpreter.get_signature_runner('decode')\n", "\n", "# 'encoded' and 'decoded' are dictionaries with all outputs from the inference.\n", "input = tf.constant([1, 2, 3], dtype=tf.float32)\n", "print('Input:', input)\n", "encoded = encode(x=input)\n", "print('Encoded result:', encoded)\n", "decoded = decode(x=encoded['encoded_result'])\n", "print('Decoded result:', decoded)" ] }, { "cell_type": "markdown", "id": "81b42e5b", "metadata": { "id": "81b42e5b" }, "source": [ "## Known limitations\n", "\n", "* As the LiteRT interpreter does not guarantee thread safety, signature runners\n", " from the same interpreter must not be executed concurrently.\n", "* Support for iOS/Swift is not available yet.\n", "\n" ] }, { "cell_type": "markdown", "id": "3032Iof6QqmJ", "metadata": { "id": "3032Iof6QqmJ" }, "source": [ "## Updates\n", "\n", "* Version 2.7\n", " - The multiple signature feature is implemented.\n", " - All the converter APIs from version two generate signature-enabled\n", " LiteRT models.\n", "* Version 2.5\n", " - Signature feature is available through the `from_saved_model` converter\n", " API." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "Signatures in LiteRT", "provenance": [], "toc_visible": true }, "id": "a1b42e5b", "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }