Monitoring your S3 File Gateway (original) (raw)

You can monitor your S3 File Gateway and associated resources in AWS Storage Gateway by using Amazon CloudWatch metrics and audit logs. You can also use CloudWatch Events to get notified when your file operations are done.

Topics

• Getting S3 File Gateway health logs with CloudWatch log groups
• Using Amazon CloudWatch metrics
• Getting notified about file operations
• Understanding gateway metrics
• Understanding file share metrics
• Understanding S3 File Gateway audit logs

Getting S3 File Gateway health logs with CloudWatch log groups

You can use Amazon CloudWatch Logs to get information about the health of your S3 File Gateway and related resources. You can use the logs to monitor your gateway for errors that it encounters. In addition, you can use Amazon CloudWatch subscription filters to automate processing of the log information in real time. For more information, see Real-time Processing of Log Data with Subscriptions in the Amazon CloudWatch User Guide.

For example, you can configure a CloudWatch log group to monitor your gateway and get notified when your S3 File Gateway fails to upload files to an Amazon S3 bucket. You can configure the group either when you are activating the gateway or after your gateway is activated and up and running. For information about how to configure a CloudWatch log group when activating a gateway, see Configure your Amazon S3 File Gateway. For general information about CloudWatch log groups, see Working with Log Groups and Log Streams in the Amazon CloudWatch User Guide.

The following is an example of an error reported by an S3 File Gateway.

{
    "severity": "ERROR",
    "bucket": "bucket-smb-share2",
    "roleArn": "arn:aws:iam::123456789012:role/amzn-s3-demo-bucket",
    "source": "share-E1A2B34C",
    "type": "InaccessibleStorageClass",
    "operation": "S3Upload",
    "key": "myFolder/myFile.text",
    "gateway": "sgw-B1D123D4",
    "timestamp": "1565740862516"
}

This error means that the S3 File Gateway is unable to upload the objectmyFolder/myFile.text to Amazon S3 because it has transitioned out of the Amazon S3 Standard storage class to either the S3 Glacier Flexible Retrieval or the S3 Glacier Deep Archive storage class.

In the preceding gateway health log, these items specify the given information:

• source: share-E1A2B34C indicates the file share that encountered this error.
• "type": "InaccessibleStorageClass" indicates the type of error that occurred. In this case, this error was encountered when the gateway was trying to upload the specified object to Amazon S3 or read from Amazon S3. However, in this case, the object has transitioned to Amazon S3 Glacier. The value of"type" can be any error that the S3 File Gateway encounters. For a list of possible errors, see Troubleshooting: File Gateway issues.
• "operation": "S3Upload" indicates that this error occurred when the gateway was trying to upload this object to S3.
• "key": "myFolder/myFile.text" indicates the object that caused the failure.
• gateway": "sgw-B1D123D4 indicates the S3 File Gateway that encountered this error.
• "timestamp": "1565740862516" indicates the time that the error occurred.

For information about how to troubleshoot the errors that may be reported by S3 File Gateway, see Troubleshooting: File Gateway issues.

Configuring a CloudWatch log group after your gateway is activated

The following procedure shows you how to configure a CloudWatch Log Group after your gateway is activated.

To configure a CloudWatch log group to work with your S3 File Gateway

1. Sign in to the AWS Management Console and open the Storage Gateway console athttps://console.aws.amazon.com/storagegateway/home.
2. In the navigation pane, choose Gateways, and then choose the gateway that you want to configure the CloudWatch log group for.
3. For Actions, choose Edit gateway information. Or, on the Details tab, under Health logs and Not Enabled, choose Configure log group to open the Edit_CustomerGatewayName_ dialog box.
4. For Gateway health log group, choose one of the following:
   • Disable logging if you don't want to monitor your gateway using CloudWatch log groups.
   • Create a new log group to create a new CloudWatch log group.
   • Use an existing log group to use a CloudWatch log group that already exists.
     Choose a log group from the Existing log group list.
5. Choose Save changes.
6. To see the health logs for your gateway, do the following:
   1. In the navigation pane, choose Gateways, and then choose the gateway that you configured the CloudWatch log group for.
   2. Choose the Details tab, and underHealth logs, choose CloudWatch Logs. The Log group details page opens in the CloudWatch console.

Using Amazon CloudWatch metrics

You can get monitoring data for your S3 File Gateway by using either the AWS Management Console or the CloudWatch API. The console displays a series of graphs based on the raw data from the CloudWatch API. The CloudWatch API can also be used through one of the AWS SDKs or Amazon CloudWatch API tools. Depending on your needs, you might prefer to use either the graphs displayed in the console or retrieved from the API.

Regardless of which method you use to work with metrics, you must specify the following information:

• The metric dimension to work with. A dimension is a name-value pair that helps you to uniquely identify a metric. The dimensions for Storage Gateway are GatewayId and GatewayName. In the CloudWatch console, you can use the Gateway Metrics view to select gateway-specific dimensions. For more information about dimensions, see Dimensions in the Amazon CloudWatch User Guide.
• The metric name, such as ReadBytes.

The following table summarizes the types of Storage Gateway metric data that are available to you.

Working with gateway and file metrics is similar to working with other service metrics. You can find a discussion of some of the most common metrics tasks in the CloudWatch documentation listed following:

• Viewing available metrics
• Getting statistics for a metric
• Creating CloudWatch alarms

Getting notified about file operations

Storage Gateway can initiate the following CloudWatch Events when your file operations are done:

• You can get notified when the gateway finishes the asynchronous uploading of your files from the file share to Amazon S3. Use the NotificationPolicy parameter to request a file upload notification. This sends a notification for each completed file upload to Amazon S3. For more information, see Getting file upload notification.
• You can get notified when the gateway finishes the asynchronous uploading of your working file set from the file share to Amazon S3. Use the NotifyWhenUploaded API operation to request a working file set upload notification. This sends a notification when all files in the working file set have been uploaded to Amazon S3. For more information, see Getting working file set upload notification.
• You can get notified when the gateway finishes refreshing the cache for your S3 bucket. When you invoke the RefreshCache operation through the Storage Gateway console or API, subscribe to the notification when the operation is complete. For more information, see Getting refresh cache notification.

When the file operation you requested is done, Storage Gateway sends you a notification through CloudWatch Events. You can configure CloudWatch Events to send the notification through event targets such as Amazon SNS, Amazon SQS, or an AWS Lambda function. For example, you can configure an Amazon SNS target to send the notification to Amazon SNS consumers such as an email or text message. For information about CloudWatch Events, see What is CloudWatch Events?

To set up CloudWatch Events notification

1. Create a target, such as an Amazon SNS topic or Lambda function, to invoke when the event you requested in Storage Gateway occurs.
2. Create a rule in the CloudWatch Events console to invoke targets based on an event in Storage Gateway.
3. In the rule, create an event pattern for the event type. The notification sent when the event matches this rule pattern.
4. Select the target and configure the settings.

The following example shows a rule that initiates the specified event type in the specified gateway and in the specified AWS Region. For example, you could specify theStorage Gateway File Upload Event as the event type.

{
   "source":[
      "aws.storagegateway"
   ],
   "resources":[
      "arn:aws:storagegateway:AWS Region:account-id
                 :gateway/gateway-id"
   ],
   "detail-type":[
      "Event type"
   ]
}

For information about how to use CloudWatch Events rules, see Creating a CloudWatch Events rule that triggers on an event in the_Amazon CloudWatch Events User Guide_.

Getting file upload notification

There are two use cases in which you can use file upload notification:

• For automating in-cloud processing of files that are uploaded, you can call the NotificationPolicy parameter and get back a notification ID. The notification that occurs when the files have been uploaded has the same notification ID as the one that was returned by the API. If you map this notification ID to track the list of files that you are uploading, you can initiate processing of the file that is uploaded in AWS when the event with the same ID is generated.
• For content distribution use cases, you can have two S3 File Gateways that map to the same Amazon S3 bucket. The file share client for Gateway1 could upload new files to Amazon S3, and the files are read by file share clients on Gateway2. The files upload to Amazon S3, but they are not visible to Gateway2 because it uses a locally cached version of files in Amazon S3. To make the files visible in Gateway2, you can use the NotificationPolicy parameter to request file upload notification from Gateway1 to notify you when the upload file is done. You can then use CloudWatch Events to automatically issue a RefreshCache request for the file share on Gateway2. When theRefreshCache request is complete, the new file is visible in Gateway2.

Example—File upload notification

The following example shows a file upload notification that is sent to you through CloudWatch when the event matches the rule you created. This notification is in JSON format. You can configure this notification to be delivered to the target as a text message. The detail-type is Storage Gateway Object Upload Event.

{
    "version": "0",
    "id": "2649b160-d59d-c97f-3f64-8aaa9ea6aed3",
    "detail-type": "Storage Gateway Object Upload Event",
    "source": "aws.storagegateway",
    "account": "123456789012",
    "time": "2020-11-05T12:34:56Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:storagegateway:us-east-1:123456789011:share/share-F123D451",
        "arn:aws:storagegateway:us-east-1:123456789011:gateway/sgw-712345DA",
        "arn:aws:s3:::do-not-delete-bucket"
    ],
    "detail": {
        "object-size": 1024,
        "modification-time": "2020-01-05T12:30:00Z",
        "object-key": "my-file.txt",
        "event-type": "object-upload-complete",        
        "prefix": "prefix/",
        "bucket-name": "amzn-s3-demo-bucket",  
    }
}               

Getting working file set upload notification

There are two use cases in which you can use the working file set upload notification:

• For automating in-cloud processing of files that are uploaded, you can call the NotifyWhenUploaded API and get back a notification ID. The notification that occurs when the working set of files have been uploaded has the same notification ID as the one that was returned by the API. If you map this notification ID to track the list of files that you are uploading, you can initiate processing of the working set of files that are uploaded in AWS when the event with the same ID is generated.
• For content distribution use cases, you can have two S3 File Gateways that map to the same Amazon S3 bucket. The file share client for Gateway1 can upload new files to Amazon S3, and the files are read by file share clients on Gateway2. The files upload to Amazon S3, but they aren't visible to Gateway2 because it uses a locally cached version of files in S3. To make the files visible in Gateway2, use the NotifyWhenUploaded API operation to request file upload notification from Gateway1, to notify you when the upload of the working set of files is done. You can then use the CloudWatch Events to automatically issue a RefreshCache request for the file share on Gateway2. When theRefreshCache request is complete, the new files are visible in Gateway2. This operation does not import files into the gateway cache storage. It only updates the cached inventory to reflect changes in the inventory of the objects in the S3 bucket.

Example—Working file set upload notification

The following example shows a working file set upload notification that is sent to you through CloudWatch when the event matches the rule you created. This notification is in JSON format. You can configure this notification to be delivered to the target as a text message. The detail-type isStorage Gateway File Upload Event.

{
    "version": "2012-10-17",
    "id": "2649b160-d59d-c97f-3f64-8aaa9ea6aed3",
    "detail-type": "Storage Gateway File Upload Event",
    "source": "aws.storagegateway",
    "account": "123456789012",
    "time": "2017-11-06T21:34:42Z",
    "region": "us-east-2",
    "resources": [
        "arn:aws:storagegateway:us-east-2:123456789011:share/share-F123D451",
        "arn:aws:storagegateway:us-east-2:123456789011:gateway/sgw-712345DA"
    ],
    "detail": {
        "event-type": "upload-complete",
        "notification-id": "11b3106b-a18a-4890-9d47-a1a755ef5e47",
        "request-received": "2018-02-06T21:34:42Z",
        "completed": "2018-02-06T21:34:53Z"
    }
}               

Getting refresh cache notification

For refresh cache notification use case, you can have two S3 File Gateways that map to the same Amazon S3 bucket and the NFS client for Gateway1 uploads new files to the S3 bucket. The files upload to Amazon S3, but they don't appear in Gateway2 until you refresh the cache. This is because Gateway2 uses a locally cached version of the files in Amazon S3. You might want to do something with the files in Gateway2 when the refresh cache is done. Large files could take a while to show up in Gateway2, so you might want to be notified when the cache refresh is done. You can request refresh cache notification from Gateway2 to notify you when all the files are visible in Gateway2.

Example—Refresh cache notification

The following example shows a refresh cache notification that is sent to you through CloudWatch when the event matches the rule you created. This notification is in JSON format. You can configure this notification to be delivered to the target as a text message. The detail-type is Storage Gateway Refresh Cache Event.

{
    "version": "2012-10-17",
    "id": "2649b160-d59d-c97f-3f64-8aaa9ea6aed3",
    "detail-type": "Storage Gateway Refresh Cache Event",
    "source": "aws.storagegateway",
    "account": "209870788375",
    "time": "2017-11-06T21:34:42Z",
    "region": "us-east-2",
    "resources": [
        "arn:aws:storagegateway:us-east-2:123456789011:share/share-F123D451",
        "arn:aws:storagegateway:us-east-2:123456789011:gateway/sgw-712345DA"
    ],
    "detail": {
        "event-type": "refresh-complete",
        "notification-id": "1c14106b-a18a-4890-9d47-a1a755ef5e47",
        "started": "2018-02-06T21:34:42Z",
        "completed": "2018-02-06T21:34:53Z",
        "folderList": [
            "/"
        ]
    }
}

Understanding gateway metrics

The following table describes metrics that cover S3 File Gateways. Each gateway has a set of metrics associated with it. Some gateway-specific metrics have the same name as certain file-share-specific metrics. These metrics represent the same kinds of measurements, but are scoped to the gateway rather than the file share.

Always specify whether you want to work with a gateway or a file share when working with a particular metric. Specifically, when working with gateway metrics, you must specify the Gateway Name for the gateway whose metric data you want to view. For more information, see Using Amazon CloudWatch metrics.

Note

Some metrics return data points only when new data has been generated during the most recent monitoring period.

The following table describes the metrics that you can use to get information about your S3 File Gateways.

Understanding file share metrics

You can find information following about the Storage Gateway metrics that cover file shares. Each file share has a set of metrics associated with it. Some file share-specific metrics have the same name as certain gateway-specific metrics. These metrics represent the same kinds of measurements, but are scoped to the file share instead.

Always specify whether you want to work with either a gateway or a file share metric before working with a metric. Specifically, when working with file share metrics, you must specify the File share ID that identifies the file share for which you are interested in viewing metrics. For more information, see Using Amazon CloudWatch metrics.

Note

Some metrics return data points only when new data has been generated during the most recent monitoring period.

The following table describes the Storage Gateway metrics that you can use to get information about your file shares.

Understanding S3 File Gateway audit logs

Amazon S3 File Gateway (S3 File Gateway) audit logs provide you with details about user access to files and folders within a file share. You can use them to monitor user activities and take action if inappropriate activity patterns are identified.

Operations

The following table describes the S3 File Gateway audit log file access operations.

Attributes

The following table describes S3 File Gateway audit log file access attributes.

Attributes logged per operation

The following table describes the S3 File Gateway audit log attributes logged in each file access operation.