Transforms with Object Lambda — MinIO Object Storage for Linux (original) (raw)

Table of Contents

MinIO’s Object Lambda enables developers to programmatically transform objects on demand. You can transform objects as needed for your use case, such as redacting personally identifiable information (PII), enriching data with information from other sources, or converting between formats.

Overview

An Object Lambda handler is a small code module that transforms the contents of an object and returns the results. Like Amazon S3 Object Lambda functions, you trigger a MinIO Object Lambda handler function with a GET request from an application. The handler retrieves the requested object from MinIO, transforms it, and returns the modified data back to MinIO to send to the original application. The original object remains unchanged.

Each handler is an independent process, and multiple handlers can transform the same data. This allows you to use the same object for different purposes without maintaining different versions of the original.

Object Lambda Handlers

You can write a handler function in any language capable of sending and receiving HTTP requests. It must be able to:

Create a Function

A handler function should perform the following steps:

  1. Extract the object details from the incoming POST request.
    The getObjectContext property of the JSON request payload contains details about the original object. To construct the response, you need the following values:
    Value Description
    inputS3Url A presigned URL for the original object. The calling application generates the URL and sends it in the original request. This allows the handler to access the original object without the MinIO credentials usually required. The URL is valid for one hour.
    outputRoute A token that allows MinIO to validate the destination for the transformed object. Return this value with the response in an x-amz-request-route header.
    outputToken A token that allows MinIO to validate the response. Return this value in the response in an x-amz-request-token header.
  2. Retrieve the original object from MinIO.
    Use the presigned URL to retrieve the object from the MinIO deployment. The contents of the object are in the body of the response.
  3. Transform the object as desired.
    Perform any operations needed to generate a transformed object. Since the calling application is waiting for a response, you may wish to avoid potentially long running operations.
  4. Construct a response containing the following information:
    • The transformed object contents.
    • An x-amz-request-route header with the outputRoute token.
    • An x-amz-request-token header with the outputToken token.
  5. Return the response back to Object Lambda.
    MinIO validates the response and sends the transformed data back to the original calling application.

Response headers

Handlers must include the outputRoute and outputToken values in the appropriate response headers. This allows MinIO to correctly validate the response from the handler.

Register the Handler

To enable MinIO to call the handler, register the handler function as a webhook with the following MinIO server Object Lambda environment variables:

MINIO_LAMBDA_WEBHOOK_ENABLE_functionname

Enable or disable Object Lambda for a handler function. For multiple handlers, set this environment variable for each function name.

MINIO_LAMBDA_WEBHOOK_ENDPOINT_functionname

Register an endpoint for a handler function. For multiple handlers, set this environment variable for each function endpoint.

MinIO also supports the following environment variables for authenticated webhook endpoints:

MINIO_LAMBDA_WEBHOOK_AUTH_TOKEN_functionanme

Specify the opaque string or JWT authorization token for authenticating to the webhook.

MINIO_LAMBDA_WEBHOOK_CLIENT_CERT_functionname

Specify the client certificate to use for mTLS authentication to the webhook.

MINIO_LAMBDA_WEBHOOK_CLIENT_KEY_functionname

Specify the private key to use for mTLS authentication to the webhook.

Restart MinIO to apply the changes.

Alternatively, configure Object Lambda with the MinIO Client command line tool. For more information, see Object Lambda function settings.

Trigger From an Application

To request a transformed object from your application:

  1. Connect to the MinIO deployment.
  2. Set the Object Lambda target by adding a lambdaArn parameter with the ARN of the desired handler.
  3. Generate a presigned URL for the original object.
  4. Use the generated URL to retrieve the transformed object.
    MinIO sends the request to the target Object Lambda handler. The handler returns the transformed contents back to MinIO, which validates the response and sends it back to the application.

Example

Transform the contents of an object using Python, Go, and curl:

Prerequisites:

Create a Handler

The sample handler, written in Python, retrieves the target object using a presigned URL generated by the caller. The handler then transforms the object’s contents and returns the new text. It uses the Flask web framework and Python 3.8+.

The following command installs Flask and other needed dependencies:

pip install flask requests

The handler calls swapcase() to change the case of each letter in the original text. It then sends the results back to MinIO, which returns it to the caller.

from flask import Flask, request, abort, make_response import requests

app = Flask(name) @app.route('/', methods=['POST']) def get_webhook(): if request.method == 'POST': # Get the request event from the 'POST' call event = request.json

  # Get the object context
  object_context = event["getObjectContext"]

  # Get the presigned URL
  # Used to fetch the original object from MinIO
  s3_url = object_context["inputS3Url"]

  # Extract the route and request tokens from the input context
  request_route = object_context["outputRoute"]
  request_token = object_context["outputToken"]

  # Get the original S3 object using the presigned URL
  r = requests.get(s3_url)
  original_object = r.content.decode('utf-8')

  # Transform the text in the object by swapping the case of each char
  transformed_object = original_object.swapcase()

  # Return the object back to Object Lambda, with required headers
  # This sends the transformed data to MinIO
  # and then to the user
  resp = make_response(transformed_object, 200)
  resp.headers['x-amz-request-route'] = request_route
  resp.headers['x-amz-request-token'] = request_token
  return resp

else: abort(400)

if name == 'main': app.run()

Start the Handler

Use the following command to start the handler in your local development environment:

The output resembles the following:

Start MinIO

Once the handler is running, start MinIO with the MINIO_LAMBDA_WEBHOOK_ENABLE and MINIO_LAMBDA_WEBHOOK_ENDPOINT environment variables to register the function with MinIO. To identify the specific Object Lambda handler, append the name of the function to the name of the environment variable.

The following command starts MinIO in your local development environment:

MINIO_LAMBDA_WEBHOOK_ENABLE_myfunction=on MINIO_LAMBDA_WEBHOOK_ENDPOINT_myfunction=http://localhost:5000 minio server /data

Replace myfunction with the name of your handler function and /data with the location of the MinIO directory for your local deployment. The output resembles the following:

MinIO Object Storage Server Copyright: 2015-2023 MinIO, Inc. License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html Version: RELEASE.2023-03-24T21-41-23Z (go1.19.7 linux/arm64)

Status: 1 Online, 0 Offline. API: http://192.168.64.21:9000 http://127.0.0.1:9000 RootUser: minioadmin RootPass: minioadmin Object Lambda ARNs: arn:minio:s3-object-lambda::myfunction:webhook

Test the Handler

To test the Lambda handler function, first create an object to transform. Then invoke the handler, in this case with curl, using the presigned URL from a Go function.

  1. Create a bucket and object for the handler to transform.
    mc alias set myminio/ http://localhost:9000 minioadmin minioadmin
    mc mb myminio/myfunctionbucket
    cat > testobject << EOF
    Hello, World!
    EOF
    mc cp testobject myminio/myfunctionbucket/
  2. Invoke the Handler
    The following Go code uses the The MinIO Go SDK to generate a presigned URL and print it to stdout.
    package main
    import (
    "context"
    "log"
    "net/url"
    "time"
    "fmt"
    "github.com/minio/minio-go/v7"
    "github.com/minio/minio-go/v7/pkg/credentials"
    )
    func main() {
    // Connect to the MinIO deployment
    s3Client, err := minio.New("localhost:9000", &minio.Options{
    Creds: credentials.NewStaticV4("my_admin_user", "my_admin_password", ""),
    Secure: false,
    })
    if err != nil {
    log.Fatalln(err)
    }
    // Set the Lambda function target using its ARN
    reqParams := make(url.Values)
    reqParams.Set("lambdaArn", "arn:minio:s3-object-lambda::myfunction:webhook")
    // Generate a presigned url to access the original object
    presignedURL, err := s3Client.PresignedGetObject(context.Background(), "myfunctionbucket", "testobject", time.Duration(1000)*time.Second, reqParams)
    if err != nil {
    log.Fatalln(err)
    }
    // Print the URL to stdout
    fmt.Println(presignedURL)
    }
    In the code above, replace the following values:
    • Replace my_admin_user and my_admin_password with user credentials for a MinIO deployment.
    • Replace myfunction with the same function name set in the MINIO_LAMBDA_WEBHOOK_ENABLE and MINIO_LAMBDA_WEBHOOK_ENDPOINT environment variables.
      To retrieve the transformed object, execute the Go code with curl to generate a GET request:
      curl -v $(go run presigned.go)
      curl runs the Go code and then retrieves the object with a GET request to the presigned URL. The output resembles the following: