Generate Extension — NVIDIA Triton Inference Server (original) (raw)
[!NOTE] The Generate Extension is provisional and likely to change in future versions.
This document describes Triton’s generate extension. The generate extension provides a simple text-oriented endpoint schema for interacting with large language models (LLMs). The generate endpoint is specific to HTTP/REST frontend.
HTTP/REST#
In all JSON schemas shown in this document, $number, $string, $boolean,$object and $array refer to the fundamental JSON types. #optional indicates an optional JSON field.
Triton exposes the generate endpoint at the following URLs. The client may use HTTP POST request to different URLs for different response behavior, the endpoint will return the generate results on success or an error in the case of failure.
POST v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/generate
POST v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/generate_stream
generate vs. generate_stream#
Both URLs expect the same request JSON object, and generate the same JSON response object. However, there are some differences in the format used to return each:
/generatereturns exactly 1 response JSON object with aContent-Typeofapplication/json/generate_streammay return multiple responses based on the inference results, with aContent-Typeoftext/event-stream; charset=utf-8. These responses will be sent asServer-Sent Events(SSE), where each response will be a “data” chunk in the HTTP response body. In the case of inference errors, responses will have an error JSON object.- Note that the HTTP response code is set in the first response of the SSE, so if the first response succeeds but an error occurs in a subsequent response for the request, it can result in receiving an error object while the status code shows success (200). Therefore, the user must always check whether an error object is received when generating responses through
/generate_stream. - If the request fails before inference begins, then a JSON error will be returned with
Content-Typeofapplication/json, similar to errors from other endpoints with the status code set to an error.
- Note that the HTTP response code is set in the first response of the SSE, so if the first response succeeds but an error occurs in a subsequent response for the request, it can result in receiving an error object while the status code shows success (200). Therefore, the user must always check whether an error object is received when generating responses through
Generate Request JSON Object#
The generate request object, identified as $generate_request, is required in the HTTP body of the POST request. The model name and (optionally) version must be available in the URL. If a version is not provided, the server may choose a version based on its own policies or return an error.
$generate_request = { "id" : $string, #optional "text_input" : $string, "parameters" : $parameters #optional }
- “id”: An identifier for this request. Optional, but if specified this identifier must be returned in the response.
- “text_input” : The text input that the model should generate output from.
- “parameters” : An optional object containing zero or more parameters for this generate request expressed as key/value pairs. SeeParameters for more information.
[!NOTE] Any additional properties in the request object are passed either as parameters or tensors based on model specification.
Parameters#
The $parameters JSON describes zero or more “name”/”value” pairs, where the “name” is the name of the parameter and the “value” is a$string, $number, or $boolean.
$parameters = { $parameter, ... } parameter=parameter = parameter=string : string∣string | string∣number | $boolean
Parameters are model-specific. The user should check with the model specification to set the parameters.
Example Request#
Below is an example to send generate request with additional model parameters stream and temperature.
$ curl -X POST localhost:8000/v2/models/mymodel/generate -d '{"id": "42", "text_input": "client input", "parameters": {"stream": false, "temperature": 0}}'
POST /v2/models/mymodel/generate HTTP/1.1 Host: localhost:8000 Content-Type: application/json Content-Length: { "id" : "42", "text_input" : "client input", "parameters" : { "stream": false, "temperature": 0 } }
Generate Response JSON Object#
A successful generate request is indicated by a 200 HTTP status code. The generate response object, identified as $generate_response, is returned in the HTTP body.
$generate_response = { "id" : $string "model_name" : $string, "model_version" : $string, "text_output" : $string }
- “id” : The “id” identifier given in the request, if any.
- “model_name” : The name of the model used for inference.
- “model_version” : The specific model version used for inference.
- “text_output” : The output of the inference.
Example Response#
200 { "id" : "42" "model_name" : "mymodel", "model_version" : "1", "text_output" : "model output" }
Generate Response JSON Error Object#
A failed generate request must be indicated by an HTTP error status (typically 400). The HTTP body must contain the$generate_error_response object.
$generate_error_response = { "error": }
- “error” : The descriptive message for the error.
Example Error#
400 { "error" : "error message" }