Monitor Private Service Connect connections (original) (raw)

This page describes how to monitor both the producer and consumer sides of Private Service Connect connections.

Private Service Connect exposes key metrics to Cloud Monitoring that give you insights into your Private Service Connect connections.

Metrics are sent automatically to Monitoring. There, you can create custom dashboards, set up alerts, and query the metrics.

For information about monitoring Private Service Connect connections that aren't supported by Private Service Connect metrics, seeLimitations.

Monitor published services

You can monitor published services by using predefined dashboards or Google Cloud metrics.

View dashboards for published services

Private Service Connect provides a set of predefined dashboards that display the following metrics for a published service:

To view predefined dashboards from the details page of a particular Private Service Connect published service, follow these steps:

Console

  1. In the Google Cloud console, go to the Private Service Connect page.
    Go to Private Service Connect
  2. Click the Published services tab.
  3. Click an existing service.
  4. Click the Monitoring tab.
    You can change the view of the charts by using the control at the top of the page. Hovering over a point on the graph gives you details for that specific time.

Metrics for published services

The "metric type" strings in this table must be prefixed with compute.googleapis.com/. That prefix has been omitted from the entries in the table.

For a full list of Google Cloud metrics, seeGoogle Cloud metrics.

For information about using these metrics for troubleshooting, seepublished service troubleshooting.

Metric type Launch stage _(Resource hierarchy levels)_Display name
Kind, Type, UnitMonitored resources _Description_Labels
private_service_connect/producer/closed_connections_count GA (project) Closed connections count
DELTA, INT64, {connection} gce_service_attachment Count of TCP/UDP connections closed over a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/connected_consumer_forwarding_rules GA (project) Connected consumer forwarding rules
GAUGE, INT64, 1 gce_service_attachment Number of Consumer Forwarding Rules connected to a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 165 seconds.
private_service_connect/producer/dropped_received_packets_count GA (project) Received packets dropped count
DELTA, INT64, {packet} gce_service_attachment Count of received packets dropped by a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/dropped_sent_packets_count GA (project) Sent packets dropped count
DELTA, INT64, {packet} gce_service_attachment Count of sent packets dropped by a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/nat_ip_address_capacity GA (project) Nat ip address capacity
GAUGE, INT64, 1 gce_service_attachment Number of total IP addresses of a PSC Service Attachment resource ID. (Value -1 means the number is larger than the max value of INT64.) Sampled every 60 seconds. After sampling, data is not visible for up to 165 seconds.
private_service_connect/producer/new_connections_count GA (project) New connections count
DELTA, INT64, {connection} gce_service_attachment Count of new TCP/UDP connections created over a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/open_connections GA (project) Open connections
GAUGE, INT64, {connection} gce_service_attachment Number of TCP/UDP connections currently open on a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/received_bytes_count GA (project) Received bytes count
DELTA, INT64, By gce_service_attachment Count of bytes received (PSC -> Service) over a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/received_packets_count GA (project) Received packets count
DELTA, INT64, {packet} gce_service_attachment Count of packets received (PSC -> Service) over a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/sent_bytes_count GA (project) Sent bytes count
DELTA, INT64, By gce_service_attachment Count of bytes sent (Service -> PSC) over a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/sent_packets_count GA (project) Sent packets count
DELTA, INT64, {packet} gce_service_attachment Count of packets sent (Service -> PSC) over a PSC Service Attachment resource ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.psc_connection_id: The Private Service Connect connection ID of the Private Service Connect Forwarding Rule.
private_service_connect/producer/used_nat_ip_addresses GA (project) Used nat ip addresses
GAUGE, INT64, 1 gce_service_attachment IP usage of the monitored service attachment. Sampled every 60 seconds. After sampling, data is not visible for up to 165 seconds.

Monitor endpoints and backends that connect to published services

This section describes how to monitor Private Service Connect endpoints and backends that connect to published services. The available options depend on the type of consumer (endpoint or backend).

This section doesn't apply to endpoints or backends that connect to Google APIs. For information about monitoring Google APIs, seeLimitations.

View dashboards for endpoints

Private Service Connect provides a set of predefined dashboards that display the following metrics for endpoints that connect to published services:

To view predefined dashboards from the details page of a particular Private Service Connect endpoint, follow these steps:

Console

  1. In the Google Cloud console, go to the Private Service Connect page.
    Go to Private Service Connect
  2. Click the Connected endpoints tab.
  3. Click an endpoint that connects to a published service.
  4. Click the Monitoring tab.
    You can change the view of the charts by using the control at the top of the page. Hovering over a point on the graph gives you details for that specific time.

Metrics for endpoints and backends

Both Private Service Connect endpoints andbackends are monitored asPrivate Service Connect Endpoint resources.

The metrics in this table are not generated for endpoints or backends that connect to Google APIs.

The "metric type" strings in this table must be prefixed with compute.googleapis.com/. That prefix has been omitted from the entries in the table.

For a full list of Google Cloud metrics, seeGoogle Cloud metrics.

For information about using these metrics to troubleshoot endpoints, seeendpoint troubleshooting.

For information about using these metrics to troubleshoot backends, seebackend troubleshooting.

Metric type Launch stage _(Resource hierarchy levels)_Display name
Kind, Type, UnitMonitored resources _Description_Labels
private_service_connect/consumer/closed_connections_count GA (project) Closed connections count
DELTA, INT64, {connection} compute.googleapis.com/PrivateServiceConnectEndpoint Count of TCP/UDP connections closed over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/dropped_received_packets_count GA (project) Received packets dropped count
DELTA, INT64, {packet} compute.googleapis.com/PrivateServiceConnectEndpoint Count of received packets dropped by a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/dropped_sent_packets_count GA (project) Sent packets dropped count
DELTA, INT64, {packet} compute.googleapis.com/PrivateServiceConnectEndpoint Count of sent packets dropped by a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/new_connections_count GA (project) New connections count
DELTA, INT64, {connection} compute.googleapis.com/PrivateServiceConnectEndpoint Count of new TCP/UDP connections created over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/open_connections GA (project) Open connections
GAUGE, INT64, {connection} compute.googleapis.com/PrivateServiceConnectEndpoint Number of TCP/UDP connections currently open on a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/received_bytes_count GA (project) Received bytes count
DELTA, INT64, By compute.googleapis.com/PrivateServiceConnectEndpoint Count of bytes received (PSC -> Clients) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/received_packets_count GA (project) Received packets count
DELTA, INT64, {packet} compute.googleapis.com/PrivateServiceConnectEndpoint Count of packets received (PSC -> Clients) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/sent_bytes_count GA (project) Sent bytes count
DELTA, INT64, By compute.googleapis.com/PrivateServiceConnectEndpoint Count of bytes sent (Clients -> PSC) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.
private_service_connect/consumer/sent_packets_count GA (project) Sent packets count
DELTA, INT64, {packet} compute.googleapis.com/PrivateServiceConnectEndpoint Count of packets sent (Clients -> PSC) over a PSC connection ID. Sampled every 60 seconds. After sampling, data is not visible for up to 345 seconds. ip_protocol: The protocol of the connection. Can be TCP or UDP.

Define alerting policies

To create a metrics-based alerting policy, follow these steps. Use a resource type of Service Attachment for metrics about published services. Use a resource type of Private Service Connect Endpoint for metrics about endpoints or backends.

Console

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

  1. In the Google Cloud console, go to theAlerting page:
    Go to Alerting
    If you use the search bar to find this page, then select the result whose subheading isMonitoring.
  2. If you haven't created your notification channels and if you want to be notified, then clickEdit Notification Channels and add your notification channels. Return to theAlerting page after you add your channels.
  3. From the Alerting page, select Create policy.
  4. To select the metric, expand the Select a metric menu and then do the following:
    1. To limit the menu to relevant entries, enter the resource type into the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle.
    2. For the Resource type, select the resource type.
    3. For the Metric category, select Private_service_connect.
    4. For the Metric, select the metric to use for this policy.
    5. Select Apply.
  5. Click Next.
  6. The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, seeCreate metric-threshold alerting policies.
  7. Click Next.
  8. Optional: To add notifications to your alerting policy, clickNotification channels. In the dialog, select one or more notification channels from the menu, and then click OK.
  9. Optional: Update the Incident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
  10. Optional: Click Documentation, and then add any information that you want included in a notification message.
  11. Click Alert name and enter a name for the alerting policy.
  12. Click Create Policy. For more information, see Alerting overview.

View logs

You can view logs for Private Service Connect endpoints and published services by using Cloud Logging. Cloud Logging is a fully managed service that lets you store, search, analyze, monitor, and alert on logging data and events.

You can use these logs to correlate events between the service consumer and service producer. For example, if the connection status of a consumer forwarding rule changes unexpectedly, you can request that the service producer verify their logs for any service attachment deletion or update events.

Console

  1. In the Google Cloud console, go to the Logs Explorer page.
    Go to Logs Explorer
  2. If you don't see the query editor field in the Query pane, click theShow query toggle.
  3. In the query editor field, enter a query. For example, to view an endpoint's connection status change, enter the following query, replacing CONSUMER_PROJECT_ID with the consumer project ID:
    resource.type="gce_forwarding_rule"
    log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
    protoPayload.methodName="LogPscConnectionStatusUpdate"
    For more examples of queries that you can run to view common logging events, see Common logging events for endpoints.
  4. Click Run query.

For more information about querying your audit logs, see Viewing audit logs.

Common logging events for published services

The following table lists common logging events for Private Service Connect published services.

Event description Logging advanced filter
Service attachment deletion resource.type="audited_resource" log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity" resource.labels.method="compute.serviceAttachments.delete"
Service attachment enabling connection reconciliation resource.type="audited_resource" log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity" resource.labels.method="compute.serviceAttachments.patch" protoPayload.request.reconcileConnections="true"
Service attachment rejecting a consumer project URI resource.type="audited_resource" log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity" protoPayload.request.consumerRejectLists="CONSUMER_PROJECT_ID"
Endpoint connection status change due to service attachment connection policy or organization policy resource.type="gce_service_attachment" log_name="projects/PRODUCER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event" protoPayload.methodName="LogPscProducerConnectionStatusChange"
VPC Flow Logs for traffic from a Private Service Connect subnet to any backend VM instance (including GKE nodes) resource.type="gce_subnetwork" logName="projects/PRODUCER_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" json_payload.connection.src_ip=~"PSC_SUBNET_REGEX.*" jsonPayload.dest_instance.vm_name=~"VM_INSTANCE_PREFIX.*"

Replace the following:

Common logging events for endpoints

The following table lists common logging events for Private Service Connect endpoints.

Event description Logging advanced filter
Private Service Connect endpoint creation resource.type="gce_forwarding_rule" log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity" protoPayload.methodName="v1.compute.forwardingRules.insert" "compute.forwardingRules.pscCreate"
Private Service Connect endpoint creation failure resource.type="gce_forwarding_rule" log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity" protoPayload.methodName="v1.compute.forwardingRules.insert" "compute.forwardingRules.pscCreate" severity>=ERROR
Private Service Connect endpoint connection status change resource.type="gce_forwarding_rule" log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event" protoPayload.methodName="LogPscConnectionStatusUpdate"
Rejected Private Service Connect endpoint connection resource.type="gce_forwarding_rule" log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event" protoPayload.methodName="LogPscConnectionStatusUpdate" protoPayload.metadata.pscConnectionStatus="REJECTED"
Quota PSC_INTERNAL_LB_FORWARDING_RULES exceeded resource.type="gce_forwarding_rule" log_name="projects/CONSUMER_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity" protoPayload.methodName="v1.compute.forwardingRules.insert" "QUOTA_EXCEEDED" severity=ERROR
VPC Flow Logs for traffic from a VM instance to a Private Service Connect endpoint resource.type="gce_subnetwork" logName="projects/CONSUMER_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" jsonPayload.connection.dest_ip="PSC_ENDPOINT_IP_ADDRESS" jsonPayload.src_instance.vm_name="VM_INSTANCE_NAME"
VPC Flow Logs for traffic from a gateway to a Private Service Connect endpoint resource.type="vpc_flow_logs_config" logName="projects/CONSUMER_PROJECT_ID/logs/networkmanagement.googleapis.com%2Fvpc_flows" jsonPayload.connection.dest_ip="PSC_ENDPOINT_IP_ADDRESS" jsonPayload.src_gateway.name="GATEWAY_NAME"

Replace the following:

Logging events for Composite Health

If a published service is configured to useComposite Health, you can use the following queries in the Logs Explorer to viewhealth state transitions. These logs are generated each time a monitored resource changes its state—for example, from HEALTHY to UNHEALTHY.

Logging is enabled by default for producer resources (health sources and composite health checks) and consumer resources (Private Service Connect NEGs that connect to published services that use Composite Health).

Event description Logging advanced filter
Producer health state transitions logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fcompositehealth"
Consumer health state transitions logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fservicehealthchecks"

Replace PROJECT_ID with your project ID.

Limitations

Private Service Connect metrics have the following limitations:

What's next