Logging and monitoring for Application Load Balancer and Cloud CDN plugins (original) (raw)

This page shows you how to configure and use Cloud Loggingand Cloud Monitoring with Service Extensions plugins for Cloud Load Balancing and Cloud CDN.

This section describes logging for Application Load Balancer plugins. Logging is possible from both the plugin perspective and the load balancer perspective.

Log messages

Service Extensions supports generating log messages during the execution of your plugin. Recording logs is disabled by default. To record logs for a plugin, enable it when you create the pluginor update it.

Plugin log records are annotated with the following contextual information:

Standard log annotations, such as timestamp and log level.
The identity of the plugin that generated the message.
The plugin callback in which the log message was generated.
A requestId trace identifier that helps determine the request log that a log message is associated with.

Logs that are pertinent to Service Extensions are in one of the following categories:

Plugin log messages
Generated by a logging call, such as info!(...) for Rust,proxywasm.LogInfo(...) for Go, or LOG_INFO for C++. Service Extensions exports these log messages to Cloud Logging. You can log request and response headers and any actions that the plugin has taken.
You can view these messages by using the networkservices.googleapis.com service.
Cloud Load Balancing log messages
You can view these messages by using the loadbalancing.googleapis.com service.

Logging from the plugin perspective

This section describes Service Extensions logging from the plugin perspective.

Enable logging for a plugin

Service Extensions supports generating log messages during the execution of your plugin. Recording logs is disabled by default.

To record logs for a plugin, enable it when youcreate the pluginor update it.

To enable logging for an existing plugin, use thegcloud service-extensions wasm-plugins update command:

gcloud service-extensions wasm-plugins update WASM_PLUGIN
--log-config=[LOG_CONFIG,...]

Replace the following:

WASM_PLUGIN: the ID or the fully qualified name of the plugin
LOG_CONFIG: logging options for the plugin. To enable logging, set the enable option to true. Then, specify the following details:
- sample-rate: the sampling rate of activity logs as a value between 0 and 1. The value 0 indicates that log messages aren't stored. The default value 1 indicates that all log messages are stored. A floating point value between 0.0 and 1.0 indicates that a percentage of log messages is stored.
- min-log-level: the minimum severity level of plugin log messages to be exported to Cloud Logging. The default value is INFO.

After you enable logging for the plugin, you can view the messages emitted by logging statements in plugin code in Cloud Logging.

To view logs, in the Google Cloud console, go to the Logs Explorer page.

View log messages for plugins

Logs can be viewed by building queriesin the Logs Explorer.

You can view plugin logs as standalone Service Extensions logs. In this view, each plugin log message is recorded in its own log record and isn't automatically associated with request log information.

These log messages are in thenetworkservices.googleapis.com/wasm_plugin_activity log with the resource type networkservices.googleapis.com/WasmPluginVersion.

The system might also add informational log messages to this log. For example, if there's a plugin failure when a plugin invocation exceeds CPU or memory limits, a message of ERROR severity is logged. Such messages can also be seen in View and filter errors.

Plugin log samples

Consider a sample Service Extensions log entry. The value ofmessage is passed to the plugin's LOG_INFO call. The severity value depends on the log level used in the plugin log call. In the labels section, the value of the API is HTTP_REQUEST_HEADER, which indicates that the logged operation is the on_http_request_headers plugin callback.

{ "insertId": "65224aac-0000-24bd-a0e1-582429bd544c@a1", "jsonPayload": { "@type": "type.googleapis.com/google.cloud.networkservices.logging.v1.WasmPluginLogEntry", "metroIataCode": "ber", "proxyRegionCode": "DE", "message": "[add_header_plugin.cc:26]::onRequestHeaders() AddHeaderStreamContext::onRequestHeaders called", "requestId": "effc0311-6716-431b-9e2a-7586835fdff1" }, "resource": { "type": "networkservices.googleapis.com/WasmPluginVersion", "labels": { "plugin_version": "prod-1", "resource_container": "projects/123456789", "location": "global", "plugin_name": "add-headers-plugin-prod-resource" } }, "timestamp": "2023-05-10T03:05:43.317015458Z", "severity": "INFO", "labels": { "networkservices.googleapis.com/operation": "HTTP_REQUEST_HEADERS" }, "logName": "projects/123456789/logs/networkservices.googleapis.com%2Fwasm_plugin_activity", "trace": "projects/123456789/traces/effc0311-6716-431b-9e2a-7586835fdff1", "receiveTimestamp": "2023-05-10T03:05:44.207265284Z" }

Limitations for logging

Plugins are limited to logging up to 16 KiB of payload data per client HTTP request. This amount is divided across multiple logging calls that are associated with a given HTTP request. The limit applies only to log message text, not to additional metadata added to the log record by Service Extensions.

For example, if an on_http_request_headers callback makes two logging calls with 4 KiB messages each, and then an on_http_response_headers callback attempts to make three logging calls with 4 KiB messages each for the same HTTP request, the third logging message is dropped. A log message is added to record the number of plugin-generated log messages that were dropped.

Logging from the load balancer perspective

This section describes Service Extensions logging from the load balancer perspective.

Enable logging on a backend service

You can enable logging for Application Load Balancer plugins while creating a service by enabling logging on the backend service that's the target of a request.

To enable logging for the target backend service, use thegcloud compute backend-services update command.

gcloud compute backend-services update BACKEND_SERVICE
--enable-logging
--logging-sample-rate=RATE
--region=REGION
--logging-optional=LOGGING_OPTIONAL_MODE
--logging-optional-fields=OPTIONAL_FIELDS

Replace the following:

BACKEND_SERVICE: the name of the backend service
RATE: a value from 0.0 through 1.0, where 0.0 means no requests are logged and 1.0 means 100% of requests are logged. The default value is 1.0. This setting is effective only when used with the enable-logging parameter. When you omitenable-logging, logging is disabled.
REGION: the region of the backend
LOGGING_OPTIONAL_MODE: enables logging for optional fields in one of these modes:
- INCLUDE_ALL_OPTIONAL includes all optional fields.
- EXCLUDE_ALL_OPTIONAL (default) excludes all optional fields.
- CUSTOM includes a custom list of optional fields.
OPTIONAL_FIELDS: a comma-separated list of optional fields when you select the CUSTOM mode

After you enable logging on the backend service, HTTP or HTTPS requests are logged by using Cloud Logging logs requests.

To view logs, in the Google Cloud console, go to the the Logs Explorer page.

Log messages for a backend service

In general, Application Load Balancer log entries contain information that is useful for monitoring and debugging your HTTP or HTTPS traffic. Log entries contain the following types of information:

Information shown in most Google Cloud logs, such as severity, project ID, project number, and timestamp as described in theLogEntry log.
HttpRequest log fields.

Request logs for HTTP and HTTPS load balancers contain a service_extension_infoobject in the load balancer log entry JSON payload with the following information:

Field	Type	Description
backend_target_name	string	Name of the backend target of the extension.
backend_target_type	string	Type of the backend target.
chain	string	Name of the extension chain within the service extension resource that matches the request.
extension	string	Name of the extension within the extension chain.
failed_open	boolean	When the extension configuration has failOpen set to true, the value true for this metric indicates that processing continued when the extension timed out or failed.Applies only to regional external Application Load Balancers, regional internal Application Load Balancers, and cross-region internal Application Load Balancers.
grpc_status	enum	The most recent status on the gRPC stream. For more information, see gRPC status codes.
per_processing_request_info	array	A list of either ProcessingRequest stats for ext_proc extensions or CheckRequest stats for ext_authz extensions that occur over the gRPC stream.
per_processing_request_info[].event_type	enum	The event type of ProcessingRequest. Can be one of these:REQUEST_HEADERS, REQUEST_BODY,RESPONSE_HEADERS, or RESPONSE_BODY.
per_processing_request_info[].latency	duration	The duration from when the first byte of the ProcessingRequest message is sent to the extension to when the last byte of theProcessingResponse message is received.
per_processing_request_info[].processing_effect	enum	The result of processing for each event in a processing request. Applies only to regional external Application Load Balancers, regional internal Application Load Balancers, and cross-region internal Application Load Balancers.Can be one of the following values: NONE: indicates that contents were not changed. NONE_FAILED_OPEN: indicates that no mutations were performed because the extension failed open. CONTENT_MODIFIED: indicates that content was changed by a successfully applied mutation request. IMMEDIATE_RESPONSE: indicates that an immediate response was sent by the extension to halt all further processing. MUTATION_REJECTED: indicates that the extension requested at least one disallowed change and further processing was discontinued. Appropriate error messages are logged. UNSPECIFIED: indicates that the effect of processing is not known.
per_processing_request_info[].processing_effect_details	string	When processing_effect is MUTATION_REJECTED, the details about why a mutation was rejected.Applies only to regional external Application Load Balancers, regional internal Application Load Balancers, and cross-region internal Application Load Balancers.
resource	string	Name of the extension resource

Monitoring

This section describes how you can use Cloud Monitoring dashboardsto view metrics for Application Load Balancer plugins that are configured using Service Extensions. You can monitor plugins from either the plugin perspective or the load balancer perspective.

Monitoring from a plugins perspective

This section describes Service Extensions monitoring from a plugins perspective.

For detailed information about Service Extensions metric types, see the Google Cloud metricspage.

View the Monitoring dashboard for Service Extensions

To view the Monitoring dashboard for Service Extensions, do the following:

In the Google Cloud console, go to the Service Extensions page.
Go to Service Extensions
Click the Plugins tab.
Click a plugin name.
On the Plugin details page, click the Monitoring tab.

On the Monitoring page, the metrics charts show information that can help you monitor plugin performance.

To view the metrics for plugin lifecycle operations, select values from theOperation filter list. By default, the values HTTP request header and HTTP response header are selected.
To view the metrics for a specific plugin version, select a value from thePlugin version filter list. By default, metrics are displayed for all versions.
To change the period for which you want to view the data, either select a predefined period from the time selector or click Custom and define a start and end time. By default, the selector is set to 1 day.

Plugin metrics for Service Extensions

You can monitor the following metrics for plugins from the Service Extensions perspective. These metrics have the prefixnetworkservices.googleapis.com/wasm_plugin/. The prefix is omitted from the entries in the table.

Metric type	Display nameKind, Type, UnitDescription
invocation_count	Wasm plugin invocation countDELTA, INT64, 1 The number of invocations of the plugin over the selected time span. Each plugin callback invocation counts as a separate plugin invocation.
invocation_latencies	Wasm plugin invocation latencyDELTA, DISTRIBUTION, us The local execution time, in milliseconds, of the plugin. The metric includes label-delineated entries for eachcallback.
cpu/usage_times	Wasm plugin normalized CPU usageDELTA, DISTRIBUTION, us{CPU} The CPU usage time of plugin invocations, in microseconds.
memory/bytes_used	Wasm plugin memory usageGAUGE, DISTRIBUTION, By Total memory allocated by Wasm plugin VMs, in bytes.

Monitoring from the load balancer perspective

This section describes Service Extensions monitoring for plugins from the load balancer perspective.

View the Monitoring dashboard for Cloud Load Balancing

Application Load Balancers export monitoring data toCloud Monitoring.

Use Monitoring metrics for the following purposes:

Evaluating a load balancer's configuration, usage, and performance
Troubleshooting problems
Improving resource utilization and user experience

To view a predefined dashboard, follow these steps:

In the Google Cloud console, go to the Dashboards overview page.
Go to Dashboards overview
In the Categories section, click GCP.
- To view a list of dashboards for all your load balancers, in theGCP Dashboards list, click the dashboard named Google Cloud Load Balancers. To view a specific load balancer's dashboard, locate the load balancer in the list and click its name.
- To view the predefined dashboards for only your load balancers, select the appropriate dashboard.

In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query the metrics through theCloud Monitoring API.

Plugin metrics for Cloud Load Balancing

You can monitor the following metrics for plugins from a Cloud Load Balancing perspective.

In Preview, you can monitor the following metrics for extensions on all types of Application Load Balancers. These metrics have the prefixnetworkservices.googleapis.com. The prefix is omitted from the entries in the following table.

The following table provides the metric type, display name, kind, type, unit, and description for each metric.

Metric type	Display nameKind, Type, UnitDescription
extension/invocation_count	Extension invocation countDELTA, INT64, 1 The number of invocations sent to the extension.
extension/invocation_latencies	Extension invocation latenciesDELTA, DISTRIBUTION, ms The distribution calculated from the latency of each extension invocation.
extension/sent_chunks_count	Extension sent chunks countDELTA, INT64, 1 Applicable only for request_body and response_body events. The number of data chunks sent to the extension.
extension/received_chunks_count	Extension received chunks countDELTA, INT64, 1 Applicable only for request_body and response_body events. The number of chunks received from the extension.
extension/failed_open_count	Extension failed invocations with fail-openDELTA, INT64, 1 The number of times that an invocation failed when the system was configured to fail open and the request was allowed to proceed.
extension/mutation_rejections_count	Extension mutation rejections countDELTA, INT64, 1 The number of invocations that requested header, body, or trailer mutations but were rejected. Rejections can occur for a variety of reasons, such as when the mutation is invalid or exceeds size limits.
extension/sent_bytes_count	Extension sent bytes countDELTA, INT64, By The number of bytes sent to the extension.
extension/received_bytes_count	Extension received bytes countDELTA, INT64, By The number of bytes received from the extension.

You can also monitor the following metrics. These metrics have the prefixloadbalancing.googleapis.com/. The prefix is omitted from the entries in the table.

Metric type	Display nameKind, Type, UnitDescription
https/backend_request_count,https/external/regional/backend_request_count,https/internal/backend_request_count	Backend request countDELTA, INT64, 1 The number of times that a plugin is called from the Application Load Balancer.
https/backend_request_bytes_count,https/external/regional/backend_request_bytes_count,https/internal/backend_request_bytes_count	Backend request bytesDELTA, INT64, By The number of bytes sent from the load balancer to the plugin.
https/backend_response_bytes_count,https/external/regional/backend_response_bytes_count,https/internal/backend_response_bytes_count	Backend response bytesDELTA, INT64, By The number of bytes received by the load balancer from the extension backend.

View plugin metrics

To view the metrics of a specific plugin, do the following:

In the Google Cloud console, go to the Metrics explorer page.
Go to Metrics explorer
In the Metric element, expand the Select a metric menu. Then, do the following:
1. From the list of resources, select the appropriate Application Load Balancer rule.
2. From metric categories list, select Https.
3. From the list of metrics, select a plugin metric.
4. Click Apply.
In the Filter element, do the following:
1. Select the backend_target_type label and set the value to WASM_PLUGIN.
2. Select the backend_target_name label and set the plugin name as the value.

For more information about load balancer metric types, see theloadbalancingsection of the Google Cloud metrics page.