Metrics overview (original) (raw)

This page describes the metrics that help you monitor the health and performance of your Cloud Data Fusion instances and pipelines. Use Cloud Monitoring to monitor these metrics. The metrics provide insights into pipeline runs, instance details, API requests, and authorization checks.

The metrics are categorized as either pipeline metrics or instance metrics:

• Pipeline metrics provide data about individual pipeline runs, such as run status, duration, latency, and data throughput.
• Instance metrics provide aggregated information about the pipelines within an instance, including service availability, the number of deployed pipelines, and API request counts.

Filter and aggregate Cloud Data Fusion pipeline and instance metrics in Monitoring using metric and monitored-resource labels. When you customize your metrics views, you can use one or both of these label types.

Cloud Data Fusion Pipeline monitored-resource labels

Filter and aggregate the metrics with the following Cloud Data Fusion Pipeline monitored-resource labels:

Pipeline metric labels

Filter and aggregate the metrics with the following Cloud Data Fusion metric labels in Monitoring:

Cloud Data Fusion Instance monitored-resource labels

Starting with Cloud Data Fusion version 6.11.1.1, the InstanceV3(datafusion.googleapis.com/InstanceV3) monitored resource is the default resource type for instance-level metrics. All new instances and instances upgraded to version 6.11.1.1 or later automatically emit metrics and logs using InstanceV3. The InstanceV3 resource structure differs from the previous version by removing the org_id and namespace labels.

By default, the emission of InstanceV2 metrics is disabled in Cloud Data Fusion version 6.11.1.1 and later. However, you can re-enable InstanceV2 emission alongside InstanceV3 using the Cloud Data Fusion REST API if you require backward compatibility for existing dashboards or queries.

You can filter and aggregate the metrics with the following_Cloud Data Fusion Instance_ monitored-resource labels.

InstanceV3 monitored-resource labels

InstanceV2 monitored-resource labels

Instance metric labels

Filter and aggregate the metrics with the following Cloud Data Fusion metric labels in Monitoring.

InstanceV3 metric labels

InstanceV2 metric labels

Manage InstanceV2 metric emission

For Cloud Data Fusion instances running version 6.11.1.1 or later,InstanceV2 metric emission is disabled by default. If you need to maintain backward compatibility with dashboards or alerts using the old format, you can re-enable InstanceV2 metrics using the Cloud Data Fusion REST API.

Enable InstanceV2 metrics

To enable the emission of InstanceV2 metrics, use theinstances.patchmethod with enable_instance_v2_metrics set to true:

  curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://datafusion.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID?updateMask=monitoring_config" \
  --data '{"monitoringConfig": {"enable_instance_v2_metrics": true}}'

Replace the following:

• PROJECT_ID: the Google Cloud project ID
• LOCATION: thelocation of your instance
• INSTANCE_ID: the ID of your Cloud Data Fusion instance

Disable InstanceV2 metrics

To disable InstanceV2 metrics and revert to the default behavior (emitting only InstanceV3 metrics), use theinstances.patchmethod with enable_instance_v2_metrics set to false:

  curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://datafusion.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID?updateMask=monitoring_config" \
  --data '{"monitoringConfig": {"enable_instance_v2_metrics": false}}'

Replace the following:

• PROJECT_ID: the Google Cloud project ID
• LOCATION: thelocation of your instance
• INSTANCE_ID: the ID of your Cloud Data Fusion instance

Migrate Cloud Monitoring queries from InstanceV2 to InstanceV3

Starting with Cloud Data Fusion version 6.11.1.1, the InstanceV3(datafusion.googleapis.com/InstanceV3) monitored resource is the default resource type for instance-level metrics. If you have existing Monitoring dashboards, charts, or alert policies that target InstanceV2, you can update them to use the InstanceV3resource type and metric paths.

To migrate your queries, follow these steps:

1. Change resource type: Update resource.type fromdatafusion.googleapis.com/InstanceV2 todatafusion.googleapis.com/InstanceV3.
2. Update metric names: Change the metric paths from .../instance/v2/... to.../instance/v3/....
3. Remove labels: Remove any filters or aggregations based onresource.labels.org_id or resource.labels.namespace, as these labels are not present in InstanceV3.

For example, if the following is your existing InstanceV2 query:

fetch datafusion.googleapis.com/InstanceV2
| metric 'datafusion.googleapis.com/instance/v2/pipelines'
| filter resource.labels.instance_id == 'my-instance'
| group_by 1m, [value_pipelines_mean: mean(value.pipelines)]
| every 1m

Update it as follows:

fetch datafusion.googleapis.com/InstanceV3
| metric 'datafusion.googleapis.com/instance/v3/pipelines'
| filter resource.labels.instance_id == 'my-instance'
| group_by 1m, [value_pipelines_mean: mean(value.pipelines)]
| every 1m

What's next

• Learn more about viewing Cloud Data Fusion metrics in Monitoring.