View and analyze logs (original) (raw)

This document provides you with an overview of the Logs Explorer in the Google Cloud console, which you can use to retrieve, view, and analyze log entries that are stored in log buckets. Viewing and analyzing individual log entries and a sequence of log entries can help you troubleshoot problems and improve observability. You can also read log data by using theLogging API and theGoogle Cloud CLI.

To perform aggregate operations on your log entries, like counting the number of log entries that contain a specific pattern, upgrade the log bucket to use Observability Analytics, and then run queries from the Observability Analyticspage of the Google Cloud console. You can continue to use the Logs Explorer to view logs in log buckets that have been upgraded to use Observability Analytics. For more information, seeObservability Analytics overview.

The remainder of this document describes how to use the Logs Explorer.

Before you begin

To get the permissions that you need to use the Logs Explorer to view log entries, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Get started

To begin using the Logs Explorer, do the following:

  1. In the Google Cloud console, go to theLogs Explorer page:
    Go to Logs Explorer
    If you use the search bar to find this page, then select the result whose subheading isLogging.
  2. Select a Google Cloud project, folder, or organization. For App Hubconfigurations, select the App Hub host project or management project.
    The log entries displayed by the Logs Explorer page depend on the following:
    • The resources searched for log entries.
    • The time-range setting.
    • Your Identity and Access Management (IAM) roles on the searched resources.
    • Your query filters the search results. For example, adding the query severity>=ERROR results in the display listing only those log entries with a severity level of at least ERROR.
      By default, the Logs Explorer page searches the resources listed in the default log scopefor log entries. When the default log scope isn't accessible, the page searches for the log entries that originate in your selected project, folder, or organization. For projects, the search results include the log entries that are routed to the project by a sink in another project, and then stored in a log bucket.
      After the Logs Explorer page opens, you canselect which resources are searched for log entries by the Logs Explorer. However, your selection applies only to your current session.

Logs Explorer interface

The Logs Explorer interface lets you display log entries, parse and analyze them, and specify query parameters. The Logs Explorer contains the following sections, which are detailed on this page:

  1. Primary toolbar
  2. Query pane
  3. Fields pane
  4. Timeline
  5. Query results pane

User interface for the Logs Explorer.

Primary toolbar in the Logs Explorer

Using the primary toolbar, you can do the following:

View recent, saved, and suggested queries

To view queries that you recently run, queries that you saved for future use, and suggested queries, click the Query library button.

In the Query library tab, you can view the following:

Set preferences for viewing log data

To customize how your logs data is presented in the query results, click thePreferences button, and select View, Format, orManage summary fields:

Configure the Time column

The Time column in the Query results pane displays the timestamps of your log entries. You can customize the Time column so that only certain parts of the timestamp are visible. This creates more horizontal space so that you can view more information in the log entry.

To select which parts of the timestamp to display, click thePreferences menu, select Format, and then select one of the following options:

Find patterns in your logs by using summary fields

Suppose you're looking through the log entries in your query results and want to quickly skim the results by a certain LogEntry field. Or perhaps you want to group your log entries by a certain field-value pair. You can add summary fields to your results, which appear as chips at the beginning of each log entry line.

The Logs Explorer offers default summary fields and custom summary fields. Default summary fields depend on your current query results, and custom summary fields let you select any field in the LogEntry.

To show or hide all summary field chips in your query results, clickPreferences button, select View, thenShow summary chips. When this option is enabled, the results are displayed in raw-text format.

To hide specific summary fields, enable Summary fields, and then clickHide summary field. To modify summary fields, do the following:

  1. Click the Preferences button, and selectManage summary fields.
  2. In the Manage summary fields dialog, you can do the following:
    • Add any custom field names to Custom summary fields.
      The summary field supports auto-completion and field correction for legal characters within quotes. For example, if you typejsonPayload.id-field, it gets changed to jsonPayload."id-field".
      You can also select any LogEntry field, regardless of whether it is suggested to you with the autocomplete function.
      To remove an existing custom summary field, click the X in its chip.
    • Turn truncation on and off for your custom summary fields.
      To shorten the display of the summary field values, use the toggle next to Truncate custom summary fields. You can choose how many characters to display before the field is truncated, and whether the beginning or the end of the field is displayed.
    • Hide or show default summary fields:
      To customize which default summary fields are shown in your query results, expand the Hide or show default summary fields menu.
  3. Click Apply.
    Your summary fields are now updated in your query results.

View logs by time range

To specify a time range for the log entries that you want to view, use the time-range selector.

To perform a forced refresh of your query results to include the current time, click Jump to now.

Stream logs

You can stream logs as Logging stores them in log buckets, or you can add a query to stream only those logs that match the query.

To stream logs based on a query, add a query in the Query pane, and then select the Stream logs button in the primary toolbar. As Logging stores the logs data in log buckets, only those logs that match the query are shown in the Query results pane. When a query isn't provided, Logging shows each log entry as it's stored.

To stop streaming, click Stop stream, or manually scroll to pause the stream.

Query pane

To build a query in the Logs Explorer, use the query pane. In the query pane, you can build and refine queries by using the following features:

After you review your query, click Run query. Logs that match your query are listed under the Query results pane. TheTimeline and Fields panes also adjust according to the query expression.

Select which resources are searched for log entries

This section describes how to view or change which resources are searched for log entries. If you enter a query, then the Logs Explorer displays only the fetched log entries which also match the query. For example, adding the query severity>=ERROR results in the display listing only those log entries with a severity level of at least ERROR.

To determine which log entries to display, the Logs Explorer page searches the resources listed in the default log scope. When the default log scope isn't accessible, the page searches for the log entries that originate in your selected project, folder, or organization. When projects are searched, the search results include the log entries that are routed to the project by a sink in another project, and then stored in a log bucket.

For your current session, you can configure which resources the Logs Explorer searches for log entries. For example, when troubleshooting, you might want to examine only those log entries that are included in alog view or that originate in a specific Google Cloud project.

The toolbar in the Query pane displays the resources searched for log entries:

When you want to change which resources are searched for log entries, do the following:

  1. Go to the toolbar of the Query pane. This toolbar displays a menu that is labeled like one of the following:
    Select the menu.
  2. In the Refine scope flyout, select the storage location, and then complete the dialog.
    The following information might help you complete the dialog:
    • When you want to view log entries routed by anaggregated sink, select Log view as the storage location. Be sure to then select all log views that store log entries that you want to view.
    • For folders and organizations, the log view menu lists all log views that store log entries that originated in those resources.
    • For projects, the log view menu lists the log views on log buckets stored by the selected project.
    • To modify the log-view menu to include log views on log buckets stored in a specific project, click Import project, and then select the project.
    • To configure a log scope, select Log scope, and then click Manage scopes.

Fields pane

The Fields pane offers a high-level summary of logs data and provides an efficient way to refine a query. The entries in this pane also list the approximate number of log entries that match the corresponding filter. To get an exact count of the number of log entries received in a time interval, use Observability Analytics or copy logs to a Cloud Storage bucket.

After you execute a query in the query-editor field, the Fields pane is populated based on the results of the query. The pane is divided into the following sections:

For the System metadata section, the following dimensions are always available:

Some dimensions are dynamically populated based on the labels in your log data. For example, you might see a Service or Application dimension:

Other dimensions, like Project ID, are listed based on your selections. For example, the Project ID dimension is listed when your query restricts the log entries to those whose resource type isKubernetes Container, or when you've selected a log view.

Refine your query

To refine your query, select a value from the Fields pane. For example, if you select Error in the Severity heading, then the query pane is updated to include severity=ERROR.

To remove a selection, click the Clear button.

Add fields to Fields pane

You can add certain LogEntry key-value pairs to the Logs fieldpane from the log entries populated in the Query results pane. For example, if you frequently filter by the value of the jsonPayload.message field, then add it to the Fields pane.

To add a field to the Fields pane, do the following:

  1. Select a log entry and click Expand.
  2. Find the field that you want to add to the panel, select the value, and in the menu, select Pin to Fields panel.
    The custom field appears in the Fields pane as a list of key-value pairs.
    If the Pin to Fields panel isn't listed, then you can't add the field to the pane. For example, you can't add the insertID field to the Fieldspane.

To remove a custom field from the Fields pane, click Remove next to the field.

Note that the following types of fields can't be added to the Fieldspane:

Timeline

With the Timeline pane, you can visualize the distribution of logs over time. The timeline regenerates when you run a query, making it easier to see trends in your logs data and troubleshoot problems.

To show or hide the Timeline pane, click Collapse timeline.

Timeline features

  1. Timeline bars: Each timeline bar represents a time range. Each bar contains a three-color breakdown for the log-severity levels captured in each bar's time range. The colors represent the following log severities:
    • Blue: Low severities such as Default, Debug, Info, and Notice.
    • Yellow: Medium severities such as Warning.
    • Red: High severities such as Error, Critical, Alert, andEmergency.
      Each bar in the Timeline features a menu with options to analyze your logs.
  2. Time controls: Let you adjust the time range of the logs you see in the Query results pane. For details on these options, seeAnalyze logs using time controls.
  3. Time range: Shows you the time range of the logs, represented by histogram bars. The timeline helps to orient you to the logs you're viewing within the larger time range of your query.

Analyze logs using time controls

You can use time controls in the timeline to help you investigate and analyze your logs data.

Adjust time controls

The timeline provides time controls that let you adjust the data that you see in the Logs Explorer:

Timeline modifications are constrained to be between the current time ("now") and up to 30 days in the past.

Scroll or zoom to time

In addition to the time controls previously listed, the timeline provides the Scroll to time and Zoom to time features to give you more in-depth control of the timeline and the data that you see in other panes in the Logs Explorer. Perhaps a particular bar in the timeline interests you based on its relative size or severity levels. You can select that bar to adjust the logs data you see in the Logs Explorer.

You can use the Scroll to time feature to browse your logs data without changing the values in the Timeline and Fields panes. When you select theScroll to time feature, the following happens:

To select the Scroll to time feature, do the following:

  1. Hold the pointer over a bar in the timeline. A pane containing summary information about the logs data for the specified time range appears.
  2. In the pane, select Scroll to time.
    Alternatively, clicking on a timeline bar is equivalent to selecting Scroll to time.

The Zoom to time feature is similar to Scroll to time, but it runs a query on your logs data based on the time range captured by a selected timeline bar. When you select the Zoom to time feature, the following happens:

To select the Zoom to time feature, do the following:

  1. Hold the pointer over a bar in the timeline. A pane containing summary information about the logs data for the specified time range appears.
  2. In the pane, select Zoom to time.

View your query results

The Query results pane displays the result of your query. This pane also includes features that help you troubleshoot your applications. For example, you can view more details for each log entry, view similar log entries, and search for patterns and strings in your log entries.

Highlight search terms, monitor and download your logs

There are various options to analyze your logs data by selecting theActions menu of the Query results toolbar:

Download logs

To get the permissions that you need to download logs, ask your administrator to grant you theLogs View Accessor (roles/logging.viewAccessor) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This role contains the logging.logEntries.download permission. You can also grant a role that contains thelogging.logEntries.list permission, which lets a principal view and download logs.

You might also be able to get these permissions with custom roles or other predefined roles.

To download your logs, do the following:

  1. In the Actions menu of the Query results pane, click Download.
  2. In the Download logs dialog, select CSV or JSON format, and then click Download.
  3. Select what to do with the log data. You have the following options:
    • Download the data to your computer.
    • Download the data to Google Drive.
    • View the data in a new tab.
      When you save a CSV and select Google Drive, you can open the file in Google Sheets.

For troubleshooting information, see Download of logs fails.

Analyze results by using Observability Analytics

You can aggregate, analyze, and chart your log data by using SQL queries in the Observability Analytics page when your project contains log buckets that are upgraded to use Observability Analytics. To go to the Observability Analytics page from the Logs Explorer, in the Query results toolbar, clickAnalyze results.

The Analyze results button is visible only when the log bucket that contains the log view you are querying has been upgraded to use Observability Analytics. For example, if the toolbar displays N log view, then the Analyze results button is shown when the log bucket for at least one of the selected log views is upgraded to use Observability Analytics. For more information, see Upgrade a log bucket to use Observability Analytics.

The following occurs when you click Analyze results:

In the Observability Analytics page, you can view your results in tabular form, or as a chart. In the Chart tab, you can select the data that is charted, customize the chart configuration, or save a chart to a custom dashboard. For information about creating charts from your query results and saving charts to a custom dashboard, seeChart query results with Observability Analytics.

For information about using SQL to write queries, seeQuery and view logs in Observability Analytics.

Summarize a log entry by using Gemini assistance

When investigating an issue, you can use Gemini to generate a summary of the log entry. To generate a summary of a log entry, do the following:

  1. Select the log entry you want to summarize, and clickExpand.
  2. In the toolbar for the log entry, clickExplain this log entry.
    Cloud Logging sends the text of the log entry to Gemini Cloud Assist with a prompt to summarize the contents of the log entry. Gemini Cloud Assist responds with a generated summary in theGemini pane. The summary is based only on the text of the log entry without any additional context.

For more information about summarizing your log entries by using Gemini Cloud Assist, seeSummarize log entries with Gemini assistance.

Troubleshoot an issue by using Gemini assistance

To get help with troubleshooting an issue, create a Gemini Cloud Assist investigation. Investigations use log and metric data to gather insights about your complex and distributed environments and they can help you understand, diagnose, and resolve issues in your Google Cloud infrastructure and applications.

To create an investigation, do the following:

  1. Select a log entry that has a severity level of at least WARNING.
  2. Go to the toolbar for the log entry and click Investigate.

You might create an investigation to determine probable causes of issues and apply Gemini Cloud Assist-recommended solutions to help you resolve these issues. For more information about investigations, see the following documentation:

View similar log entries

You can view log entries that are similar to a selected log entry, which lets you focus on logs of interest.

To show similar log entries, do the following:

  1. Select a log entry and click Expand.
  2. Click Similar entries, and select Show similar entries.
    The query updates with a query similar to the following and reloads the query results:
    --Show similar entries
    protoPayload.methodName="io.k8s.core.v1.configmaps.update"
    --End of show similar entries

To see a preview of the similar log entries, do the following:

  1. Select a log entry and click Expand.
  2. Expand the Similar entries menu, and then selectPreview similar entries.
    A separate dialog opens with the following information:
    • The pattern that was found
    • The percentage of log entries that contain the pattern
    • Example log entries that contain the pattern
      In this dialog, you can hide or show log entries:
      Preview log entries dialog lets you hide or show similar log entries.

Hide similar log entries

You can hide similar log entries, which lets you remove logs from your query results.

There are two ways to hide similar log entries:

View or hide log entries that match a field

You can view or hide log entries that match a field in a log entry, which lets you focus on entries that contain the same field content.

To view or hide log entries that match a specific field in a log entry, do the following:

  1. Select a log entry and click Expand.
  2. Click a field's value within the log entry, such as compute.googleapis.com, which is a serviceName.
  3. From the menu, select Show matching entries or Hide matching entries.
    The query updates with a query that shows or hides similar entries, and theQuery results reload with new results.

Pin log entries

After you run a query, you can highlight a log entry by pinning it. The pinned log entry stays centered in the Query results pane. If you run a new query and the pinned log entry isn't included, then you are prompted to unpin the log entry.

To pin a log entry, do the following:

  1. Hold the pointer over the log entry that you want to pin.
  2. Click Pin.

After you pin a log entry, its background is darkened, and aPin icon is shown. A pin icon also appears on the Timeline pane based on the pinned log entry'stimestamp.

To unpin a log entry, click the pin icon again.

Show logs that match the resource of a pinned log entry

After you pin a log entry, you can run a new query that displays log entries that match the resource type or resource labels of the pinned log.

To pin a log entry and display log entries that match the same resource type or resource labels, do the following:

  1. Click Down arrownext to the pinned log to expand the pin menu.
  2. Make a selection from the pin menu:
    • To rerun the query with the sameresource.type as the pinned log, select Same resource.type.
      For example, suppose you pin a log entry with a resource.type ofk8s_node. If you select Same resource.type, then the query is rerun to display all log entries with resource.type="k8s_node".
    • To rerun the query with the sameresource.labels as the pinned log, select Same resource.labels.
    • To rerun the query with the sametrace as the pinned log, select Same trace.
    • To clear the query and show all log entries, select Show all.

View a pinned log entry in the Timeline

You can use the Timeline to highlight, scroll to, and further examine a pinned log entry.

Using the Timeline, click Pin, and then choose from the following menu options:

View trace data

When a log entry contains both the trace and the latency-related field, both the latency and trace icon appear.

Log entry display that contains trace data.

When a log entry contains only the trace field, then only the trace icon appears.

Log entry display that contains only the trace field has trace icon.

To view the trace data related to the log entry, click the trace icon. You have the following options:

View Monitoring data

For certain logs, such as GKE and Compute Engine logs, you can select the resource type from the log's summary line to display a menu with the following options:

Display of Logs Explorer menu for GKE resources.

Copy a link to a log entry

To share a link to a log, expand a log entry, and then selectCopy. You can choose to copy a log in JSON, or a link to the log. The link is copied to your clipboard. You can send the link to users who have access to the Google Cloud project. When a user pastes the link into a browser or selects it, Logging pins the log entry in their Query results pane.

Example Logs Explorer queries

For suggested queries, arranged by Google Cloud product and use case, seeSample queries using the Logs Explorer. For example, you can runKubernetes-related queries to find Google Kubernetes Engine logs.

View Compute Engine logs

For certain Compute Engine resource types, such as gce_instance andgce_network, you see the resource name with the resource ID as subtext in several places in the Logs Explorer. For example, for the gce_instanceresource type, you see the VM name alongside the VM ID. The resource names help you identify the correct resource ID, on which you can build queries.

You might see Compute Engine resource names in the following places:

Troubleshoot logging issues

This section provides instructions for troubleshooting common issues when using the Logs Explorer.

If you're experiencing issues when trying to view logs in sink destinations, see Troubleshoot routing and sinks.

You open the Logs Explorer page, and the All resources andAll log names menus are disabled.

This is expected behavior when the Logs Explorer page is searching a log view that contains a flexible filter. These filters use an extended set ofLogging query language such as disjunctive clauses.

Download of logs fails

You are using the Logs Explorer and clickDownload. The command starts but then fails to complete or reports an error.

To resolve this issue, reduce the time it takes for the command to execute by trying the following:

Can't find console logs for a VM instance

Logs written to the console of a Compute Engine instance might not be available in the Logs Explorer. To view these logs, do the following:

  1. In the Google Cloud console, go to the VM instances page:
    Go to VM instances
    If you use the search bar to find this page, then select the result whose subheading isCompute Engine.
  2. Select the instance, and then select Serial port 1 (console) in theLogs section of the page.

You can configure your instances such that their serial port output is sent to Cloud Logging. For more information, seeEnabling and disabling serial port output logging.

Get Google Cloud project or organization ID

To get a Google Cloud project or organization ID from anywhere in the Google Cloud console, expand the list of Google Cloud projects from the Google Cloud project and organization selector and find the Google Cloud project ID in the ID column.

Can't see log entries

If you don't see any log entries, check the following:

My query is correct but I still don't see log entries

Query returns an error

If you issue a query over a resource without specifying a bucket, then Cloud Logging uses the history of the sinks in the Google Cloud project to determine where entries might have been written for that resource. If Cloud Logging identifies more than 200 buckets where entries might have been written, then the query fails with the messageError: Invalid query.

To resolve this issue, refine the scope of your query to a subset of the storage. For more information, seeRefine scope.

Query results time range doesn't match query

The logs data you see in the Query results and Fields pane adjusts according to the time range captured by the histogram timeline. You adjust the histogram timeline using the histogram's time controls or thetime-range selector. Adjusting these time controls doesn't alter the query expression in the Query pane.

When you have a query with a timestamp, the time-range selector is disabled. The query uses the timestamp expression as its time-range restriction. If a query doesn't use a timestamp expression, then the query uses the time-range selector as its time-range restriction.

Get support

For information on getting support, see theGoogle Cloud Observability support page.