VPC Flow Logs (original) (raw)

VPC Flow Logs samples packets in your Virtual Private Cloud (VPC) network to generate flow logs. Flow logs are aggregated by IP connection (5-tuple). VPC Flow Logs samples the following packets:

• Packets that are sent from and received byvirtual machine (VM) instances, including instances used as Google Kubernetes Engine nodes
• Packets that are sent from and received by Cloud Run resources configured with Direct VPC egress
• Packets that are sent through VLAN attachments forCloud Interconnectand Cloud VPN tunnels

You can view flow logs in Cloud Logging, and you can export logs to any destination that Cloud Logging export supports. These logs can be used for network monitoring, forensics, security analysis, and expense optimization. For more information, seeSupported configurations.

Use cases

The following are use cases for VPC Flow Logs.

Network monitoring

VPC Flow Logs provides you with visibility into network throughput and performance. You can:

• Monitor the VPC network
• Perform network diagnosis
• Filter the flow logs by VMs, serverless endpoints, VLAN attachments, and Cloud VPN tunnels to understand traffic changes
• Understand traffic growth for capacity forecasting

Understanding network usage and optimizing network traffic expenses

You can analyze network usage with VPC Flow Logs to optimize network traffic expenses. For example, you can analyze the network flows for the following:

• Traffic between regions and zones
• Traffic to specific countries on the internet
• Traffic to on-premises and other cloud networks
• Top talkers in the network, including VMs, serverless endpoints, VLAN attachments, and Cloud VPN tunnels

Network forensics

You can use VPC Flow Logs for network forensics. For example, if an incident occurs, you can examine the following:

• Which IP addresses have communicated with each other and when
• Which IPs are compromised by analyzing all the incoming and outgoing network flows

Learn how Google Cloud users strengthen their cyber defense with VPC Flow Logs.

Supported configurations

You can enable VPC Flow Logs at the organization and project levels. An organization-level VPC Flow Logs configuration enables flow logs for all subnets, VLAN attachments, and Cloud VPN tunnels in all VPC networks in the organization.

At the project level, you can enable VPC Flow Logs for specific VPC networks, subnets, VLAN attachments, and Cloud VPN tunnels.

You can use filtering to customize these configuration scopes. For more information, see Log sampling and processing.

Logs collection

Packets are sampled within an aggregation interval. All packets collected for a given IP connection within the aggregation interval are aggregated into a single flow log entry. This data is then sent toLogging in the Google Cloud project of the VPC network that reported the flow.

Logs are stored in Logging for 30 days by default. If you want to keep logs longer than that, you can either set a custom retention period orexport them to a supported destination.

Log sampling and processing

To generate flow logs, VPC Flow Logs samples packets in your VPC network, including packets that are sent from and received by VMs and serverless endpoints and packets that pass through gateways such as VLAN attachments or Cloud VPN tunnels. After the flow logs are generated, VPC Flow Logs processes them by following the procedure described in this section.

VPC Flow Logs samples packets using a primary sampling rate. The primary sampling rate is dynamic and varies depending on the load of the physical host running the reporting resource at the time of sampling. The probability of sampling any single IP connection increases with the volume of packets. You can't control the primary flow log sampling process or adjust the primary sampling rate.

After the flow logs are generated, VPC Flow Logs processes them according to the following procedure:

1. Filtering. You can specify that only logs that match specified criteria are generated. For example, you can filter so that only logs for a particular VM or only logs with a particular metadata value are generated and the rest are discarded. For more information, seeLog filtering.
2. Aggregation. Information for sampled packets is aggregated over a configurable aggregation interval to produce a flow log entry.
3. Secondary flow log sampling. This is a second sampling process. Flow log entries are further sampled according to a configurable secondary sampling rate parameter. The secondary sampling is performed on the flow logs generated by the primary flow log sampling process. For example, if the secondary sampling rate is set to 1.0, or 100%, VPC Flow Logs samples 100% of the flow logs generated by the primary flow log sampling.
4. Metadata. If disabled, all metadata annotations are discarded. If you want to keep metadata, you can retain all fields or a specific set of fields. For more information, see Metadata annotations.
5. Write to Logging. The final log entries are written to Cloud Logging.

Because VPC Flow Logs doesn't capture every packet, it compensates for missed packets by interpolating from the captured packets. This happens for packets missed because of initial and user-configurable sampling settings.

Even though Google Cloud doesn't capture every packet, log record captures can be quite large. You can balance your traffic visibility and storage cost needs by adjusting the following aspects of logs collection:

• Aggregation interval. Sampled packets for a time interval are aggregated into a single log entry. This time interval can be 5 seconds (default), 30 seconds, 1 minute, 5 minutes, 10 minutes, or 15 minutes.
• Secondary sampling rate.
  • For configurations created with the Compute Engine API, 50% of log entries are kept by default. You can set this parameter from 1.0 (100%, all log entries are kept) to 0.0 (0%, no logs are kept).
  • For configurations created with the Network Management API, 100% of log entries are kept by default. You can set this parameter from 1.0 to greater than 0.0.
• Metadata annotations. By default, flow log entries are annotated with metadata information, such as the names of the source and destination within Google Cloud or the geographic region of external sources and destinations. Metadata annotations can be turned off, or you can specify only certain annotations, to save storage space.
• Filtering. By default, logs are generated for every sampled flow. You can set filters to generate logs that only match certain criteria.

Specifications

• VPC Flow Logs introduces no delay or performance penalty when enabled.
• VPC Flow Logs works with VPC networks, not legacy networks.
• VPC Flow Logs samples TCP, UDP, ICMP, ESP, GRE, and RDMA flows:
  • Both inbound and outbound flows are sampled. For RDMA over Converged Ethernet (RoCE), only outbound flows are sampled.
  • Flows can be within Google Cloud or between Google Cloud and other networks.
  • If a flow is captured by sampling, VPC Flow Logs generates a log for the flow. Each flow record includes the information described in theRecord format section.
  • Flow logs don't indicate which endpoint initiated the flow; they identify packet direction.
• VPC Flow Logs interacts with firewall rules in the following ways:
  • Egress packets are sampled before egress firewall rules. Even if an egress firewall rule denies outbound packets, those packets can be sampled by VPC Flow Logs.
  • Ingress packets are sampled after ingress firewall rules. If an ingress firewall rule denies inbound packets, those packets aren't sampled by VPC Flow Logs.
• You can use filters in VPC Flow Logs to generate only certain logs.
• VPC Flow Logs supports VMs that have multiple network interfaces. In each VPC, you need to enable VPC Flow Logs for each subnet that contains a network interface.
• To log flows between Pods on the same Google Kubernetes Engine (GKE) node, you must enableintranode visibilityfor the cluster.
• VPC Flow Logs isn't supported for subnets with purposeINTERNAL_HTTPS_LOAD_BALANCER because these subnets are used as proxy-only subnets and have no VM instances or serverless endpoints.
• VPC Flow Logs writes logs to the project of the reporting VPC network. For resources in Shared VPC networks, logs are reported in the host project.

Pricing and billing

Standard pricing for Logging, BigQuery, or Pub/Sub apply. VPC Flow Logs pricing is described inNetwork Telemetry pricing.

VPC Flow Logs charges are billed to the Google Cloud project of the resource that reports flow logs. If VPC Flow Logs is enabled for an organization, each project is billed separately.

What's next

• To learn more about the VPC Flow Logs record format and which metadata annotations are available, seeAbout VPC Flow Logs records.
• To see examples of VPC Flow Logs that are collected for various use cases, see About traffic flows.
• To start reporting flows for a subnet, seeConfigure VPC Flow Logs.