Multiple network interfaces (original) (raw)

This page provides an overview of multiple network interfaces for Compute Engine VM instances. Instances with multiple network interfaces are referred to as multi-NIC instances.

An instance always has at least one virtual network interface (vNIC). Depending on the machine type, you can configure additional network interfaces.

Use cases

Multi-NIC instances are useful in the following scenarios:

• To connect to resources in separate VPC networks: multi-NIC instances can connect to resources located in different VPC networks that aren't connected to each other through VPC Network Peering or NCC.
  • Because each interface of a multi-NIC instance is in a separate VPC network, you can use each interface for a unique purpose. For example, you can use some interfaces to route packets between VPC networks that carry production traffic and another interface for management or configuration purposes.
  • Within the guest OS of each multi-NIC instance, you must configure route policies and local route tables.
• Routing packets between VPC networks: multi-NIC instances can be used as next hops for routes to connect two or more VPC networks.
  • Software running within the guest OS of a multi-NIC instance can perform packet inspection, network address translation (NAT), or another network security function.
  • When connecting VPC networks using multi-NIC instances, it's a best practice to configure two or more multi-NIC instances, using them as backends for an internal passthrough Network Load Balancer in each VPC network. For more information, see Use casesin the Internal passthrough Network Load Balancers as next hops documentation.

You can also use multi-NIC instances withPrivate Service Connect interfacesto connect service producer and consumer networks in different projects.

Network interface types

Google Cloud supports the following types of network interfaces:

• vNICs: the virtual network interfaces of Compute Engine instances. Each instance must have at least one vNIC. vNICs in regular VPC networks can be either GVNIC, VIRTIO_NET, or IDPF. You can only configure vNICs when creating an instance.
• Dynamic NICs: a child interface of a parent vNIC. You can configure Dynamic NICs when you create an instance, or add them later. For more information, see Dynamic NICs.

You can also configure multi-NIC instances using machine types that include RDMA network interfaces (IRDMAor MRDMA), which must be attached to a VPC network with anRDMA network profile. Other network interface types, including Dynamic NICs, aren't supported in VPC networks with an RDMA network profile.

Specifications

The following specifications apply to instances with multiple network interfaces:

• Instances and network interfaces: every instance has a nic0 interface. The maximum number of network interfaces varies depending on the instance's machine type.
  • Each interface has an associated stack type, which determines the supported subnet stack types and IP address versions. For more information, seeStack type and IP addresses.
• Network and subnetwork requirements: each network interface must use a unique subnet in a VPC network.
  • Attaching network interfaces to subnets in different VPC networks is supported.
  • Attaching network interfaces to different subnets in the same VPC network is supported in the following scenarios:
    * For regular VPC networks, multiple network interfaces can use the same VPC network as the nic0 interface. For more information, see Multiple network interfaces in the same VPC network(Preview).
  • For VPC networks created with an RDMA network profile, multiple RDMA NICs can use the same VPC network. For more information, see the table for each network profile inSupported and unsupported features.
  • A VPC network and subnet must exist before you can create an instance whose network interface uses the network and subnet. For more information about creating networks and subnets, see Create and manage VPC networks.
• Project of instance and subnets: for multi-NIC instances in standalone projects, each network interface must use a subnet located in the same project as the instance.
  • For instances in Shared VPC host or service projects, seeShared VPC.
  • Private Service Connect interfaces provide a way for a multi-NIC instance to have network interfaces in subnets in different projects. For more information, see About network attachments.
• IP forwarding, MTU, and routing considerations: multi-NIC instances require careful planning for the following instance and interface specific configuration options:
  • The IP forwarding option is configurable on a per instance basis, applying to all network interfaces. For more information, see Enable IP forwarding for instances.
  • Each network interface can use a unique maximum transmission unit (MTU), matching the MTU of the associated VPC network. For more information, see Maximum transmission unit.
  • Each instance receives a default route using DHCP Option 121, as defined by RFC 3442. The default route is associated with nic0. Unless manually configured otherwise, any traffic leaving an instance for any destination other than a directly connected subnet will leave the instance using the default route on nic0.
    On Linux systems, you can configure custom rules and routes within the guest OS using the /etc/iproute2/rt_tablesfile and the ip rule and ip route commands. For more information, consult the guest OS documentation. For an example, see the following tutorial: Configure routing for an additional interface.

Multiple network interfaces in the same VPC network

You can attach additional vNICs of an instance to the same VPC network as the nic0 vNIC of the instance.

You might want to attach multiple vNICs to the same VPC network to increase the bandwidth of an instance in the VPC network:

• For some machine types in Google Cloud, instances are backed by multiple physical NICs on the host server, such as someGPU machine types.
  • Creating an instance with multiple vNICs where each vNIC uses a different physical NIC increases the network bandwidthof the instance for the VPC network that the vNICs are attached to.
  • When you create the instance, Compute Engine maps each vNIC to a unique physical NIC, up to the total physical NIC count of the instance's machine type.
• For most machine types in Google Cloud, instances are backed by one physical NIC. For these machine types, we recommend that you attach each vNIC to a separate VPC network.

Limitations for multiple network interfaces in the same VPC network

See the following limitations for attaching multiple network interfaces of an instance to the same VPC network:

• The additional network interfaces must be attached to the same network as thenic0 interface.
• To get additional network bandwidth for the non-nic0 interfaces, you must use a machine type that is backed by multiple physical NICs.
• The following are not supported:
  • Dynamic NICs.
  • Private Service Connect interfaces.
  • For Cloud Load Balancing, instances with multiple vNICs attached to the same VPC network can be added to instance group or zonal network endpoint group (NEG) backends. However, from the set of vNICs attached to the same VPC network, traffic can be distributed to only nic0.
  • For static routes, you can't send packets to the non-nic0 interfaces by using a next hop instance specified by name and zone (next-hop-instance). Instead, use a next hop instance specified by address (next-hop-address). For more information, seeNext hops and features.
  • Network interface bonding with link aggregation.

Dynamic NICs

If your use case requires any of the following, use Dynamic NICs. Ensure that you also review the properties andlimitations of Dynamic NICs.

• You need to add or remove network interfaces to or from existing instances. Adding or removing Dynamic NICs doesn't require restarting or recreating the instance.
• You need more network interfaces. The maximum number of vNICs for most machine types in Google Cloud is 10; however, you can configure up to 16 total interfaces by using Dynamic NICs. For more information, see Maximum number of network interfaces.
• You need to configure multi-NIC Compute Engine bare metal instances, which only have one vNIC.

Properties of Dynamic NICs

See the following information about the properties of Dynamic NICs:

• Dynamic NICs are VLAN interfaces that use the IEEE 802.1Q standard packet format. See the following considerations:
  • The VLAN ID of a Dynamic NIC must be an integer from 2 to 255.
  • The VLAN ID of a Dynamic NIC must be unique within a parent vNIC. However, Dynamic NICs that belong to different parent vNICs can use the same VLAN ID.
• Google Cloud uses the following format for the name of a Dynamic NIC:nicNUMBER.VLAN_ID, where
  • nicNUMBER is the name of the parent vNIC, such as nic0.
  • VLAN_ID is the VLAN ID that you set, such as 4.
    An example Dynamic NIC name is nic0.4.
• Creating an instance with Dynamic NICs or adding Dynamic NICs to an existing instance requires additional steps to install and manage the corresponding VLAN interfaces in the guest OS. You can use one of the following methods:
  • Configure automatic management of Dynamic NICs by using the guest agent.
  • Configure the guest OS manually.
    For more information, see Configure the guest OS for Dynamic NICs.
• Dynamic NICs share the bandwidth of their parent vNIC. To prevent any of the network interfaces from consuming all of the bandwidth, you must create an application-specific traffic policy in the guest OS to prioritize or distribute traffic, such as by using Linux Traffic Control (TC).
• Dynamic NICs share the samereceive and transmit queuesas their parent vNIC. To configure a network interface to use different receive and transmit queues, use vNICs instead of Dynamic NICs.
• The stack type of a Dynamic NIC can be the same as or different than its parent vNIC. For example, you can create IPv6-only and dual-stack Dynamic NICs under an IPv4-only parent vNIC.

Limitations of Dynamic NICs

See the following limitations of Dynamic NICs:

• You can't modify the following properties of a Dynamic NIC after it is created:
  • The parent vNIC to which the Dynamic NIC belongs.
  • The VLAN ID of the Dynamic NIC.
• Attaching multiple network interfaces of an instance to the same VPC network is not supported for Dynamic NICs.
• A Dynamic NIC can't be deleted if the Dynamic NIC is a load-balanced network interface.
• Dynamic NICs don't support the following:
  • Advanced network DDoS protectionand network edge security policiesfor Google Cloud Armor
  • Configuring IP addresses by using per-instance configurations for MIGs.
  • Features that rely on packet intercept, such as firewall endpoints
  • Compute Engine Windows drivers
• Dynamic NICs aren't supported for use withGPU instances. For more information, seeCreation errors and decreased performance when using Dynamic NICs with GPU instances.
• Dynamic NICs require that you manually create local routes in certain scenarios. For more information, seeDropped packets when using Dynamic NICs with alias IP ranges, protocol forwarding, or Passthrough Network Load Balancers.

Stack types and IP addresses

When you create a network interface, you specify one of the following interface stack types:

• IPv4-only
• Dual-stack
• IPv6-only

A VM instance can have network interfaces that have different stack types.

The following table describes supported subnet stack types and IP address details for each interface stack type:

Changing network interface stack type

You can change the stack type of a network interface as follows:

• You can convert an IPv4-only interface to dual-stack if the interface's subnet is a dual-stack subnet or if you stop the instance and assign the interface to a dual-stack subnet.
• You can convert a dual-stack interface to IPv4-only.

You can't change the stack type of an IPv6-only interface. IPv6-only interfaces are only supported when creating instances.

IPv4 address details

Each IPv4-only or dual-stack network interface receives a primary internal IPv4 address. Each interface optionally supports alias IP ranges and an external IPv4 address. The following are the IPv4 specifications and requirements:

• Primary internal IPv4 address: Compute Engine assigns the network interface a primary internal IPv4 address from the primary IPv4 address range of the interface's subnet. The primary internal IPv4 address is allocated by DHCP.
  • You can control which primary internal IPv4 address is assigned byconfiguring a static internal IPv4 addressor by specifying a custom ephemeral internal IPv4 address.
  • Within a VPC network, the primary internal IPv4 address of each VM network interface is unique.
• Alias IP ranges: optionally, you can assign the interface one or more alias IP ranges. Each alias IP range can come from either the primary IPv4 address range or a secondary IPv4 address range of the interface's subnet.
  • Within a VPC network, each interface's alias IP range must be unique.
• External IPv4 address: optionally, you can assign the interface an ephemeral or reserved external IPv4 address. Google Cloud ensures the uniqueness of each external IPv4 address.

IPv6 address details

Compute Engine assigns each dual-stack or IPv6-only network interface a /96 IPv6 address range from the/64 IPv6 address range of the interface's subnet:

• Whether the /96 IPv6 address range is internal or external depends on the_IPv6 access type_ of the interface's subnet. Google Cloud ensures the uniqueness of each internal and external IPv6 address range. For more information, see IPv6 specifications.
  • If an instance needs both an internal IPv6 address range and an external IPv6 address range: you must configure two dual-stack interfaces, two IPv6-only interfaces, or one dual-stack interface and one IPv6-only interface. The subnet used by one interface must have an external IPv6 address range, and the subnet used by the other interface must have an internal IPv6 address range.
• The first IPv6 address (/128) is configured on the interface by DHCP. For more information, see IPv6 address assignment.
• You can control which /96 IPv6 address range is assigned by configuring a static internalor externalIPv6 address range. For internal IPv6 addresses, you can specify a custom ephemeral internal IPv6 address.

If you are connecting an instance to multiple networks by using IPv6 addresses, install google-guest-agent version20220603.00or later. For more information, see I can't connect to a secondary interface's IPv6 address.

Maximum number of network interfaces

For most machine types, the maximum number of network interfaces that you can attach to an instance scales with the number of vCPUs as described in the following tables.

The following are machine-specific exceptions:

• Compute Engine bare metal instances support a single vNIC.
• The maximum number of vNICs is different for some accelerator optimized machine types, such as A3, A4, and A4X. For more information, see Accelerator-optimized machine family.

Max interface numbers

Use the following table to determine how many network interfaces can be attached to an instance.

Reference formulas

The following table provides the formulas used to calculate the maximum number of network interfaces for an instance. The formula depends on the number of vCPU.

Example distributions of Dynamic NICs

You don't have to distribute Dynamic NICs evenly across vNICs. However, you might want an even distribution because Dynamic NICs share the bandwidth of their parent vNIC.

An instance must have at least one vNIC. For example, an instance that has 2 vCPUs can have one of the following configurations:

• 1 vNIC
• 2 vNICs
• 1 vNIC and 1 Dynamic NIC

The following tables provide example configurations that evenly distribute Dynamic NICs across vNICs while using the maximum number of network interfaces for a given number of vCPU.

2 vCPUs, 2 NICs

The following table provides examples for an instance with 2 vCPUs that show how many Dynamic NICs you can have for a given number of vNICs.

4 vCPUs, 4 NICs

The following table provides examples for an instance with 4 vCPUs that show how many Dynamic NICs you can have for a given number of vNICs.

8 vCPUs, 8 NICs

The following table provides examples for an instance with 8 vCPUs that show how many Dynamic NICs you can have for a given number of vNICs.

14 vCPUs, 12 NICs

The following table provides examples for an instance with 12 vCPUs that show how many Dynamic NICs you can have for a given number of vNICs.

22 vCPUs, 16 NICs

The following table provides examples for an instance with 22 vCPUs that show how many Dynamic NICs you can have for a given number of vNICs.

Product interactions

This section describes interactions between multi-NIC instances and other products and features in Google Cloud.

Shared VPC

Except for Private Service Connect interfaces, the subnet and project relationship of a multi-NIC instance in a Shared VPC host or service project is as follows:

• Each network interface of a multi-NIC instance located in a Shared VPC host project must use a subnet of a Shared VPC network in the host project.
• Each network interface of a multi-NIC instance located in a Shared VPC service project can use either of the following:
  • A subnet of a VPC network in the service project.
  • A subnet of a Shared VPC network in the host project.

For more information about Shared VPC, see:

• Shared VPC overview
• Provision Shared VPC

Compute Engine internal DNS

Compute Engine creates internal DNS name A and PTR records only for the primary internal IPv4 address of the nic0 network interface of an instance. Compute Engine doesn't create internal DNS records for any IPv4 or IPv6 address associated with a network interface different from nic0.

For more information, seeCompute Engine internal DNS.

Static routes

Static routes can be scoped to specific instances by using network tags. When a network tag is associated with an instance, the tag applies to all network interfaces of the instance. Consequently, adding a network tag to or removing a network tag from an instance might change which static routes apply to any of the instance's network interfaces.

Load balancers

Instance group backends and zonal NEG backends each have an associated VPC network as follows:

• For managed instance groups (MIGs), the VPC network for the instance group is the VPC network assigned to the nic0interface in the instance template.
• For unmanaged instance groups, the VPC network for the instance group is the VPC network used by the nic0 network interface of the first instance that you add to the unmanaged instance group.

The following table shows which backends support distributing connections or requests to any network interface.

1Throughout the table, Any NIC doesn't apply when multiple network interfaces of an instance are in the same VPC network (Preview). For more information, seeLimitations for multiple network interfaces in the same VPC network.

Target pool-based External passthrough Network Load Balancers don't use instance groups or NEGs and only support load balancing to nic0 network interfaces.

Firewall rules

The set of firewall rules—from hierarchical firewall policies, global network firewall policies, regional network firewall policies, and VPC firewall rules—are unique to each network interface. Ensure that each network has appropriate firewall rules to allow the traffic that you want to allow to and from a multi-NIC instance. To determine which firewall rules apply to a network interface, and the source for each rule, see Get effective firewall rules for a VM interface.

Firewall rules can be scoped to specific VM instances by using network tags or secure tags, both of which apply to all network interfaces of an instance. For more information, seeComparison of secure tags and network tags.

Known issues

This section describes known issues related to using multiple network interfaces in Google Cloud.

Creation errors and decreased performance when using Dynamic NICs with GPU instances

Dynamic NICsaren't supported for use with GPU instances. If you create a GPU instance with Dynamic NICs, or add Dynamic NICs to an existing GPU instance, the following issues might occur:

• The operation fails with an error such as the following:
  Internal error. Please try again or contact Google Support. (Code: 'CODE')
• The operation succeeds, but the instance experiences decreased performance, such as significantly lower network bandwidth.

These issues occur because the Dynamic NIC configuration leads to errors when Compute Engine attempts to distribute the instance's vNICs across physical NICs on the host server.

Dropped packets when using Dynamic NICs with alias IP ranges, protocol forwarding, or Passthrough Network Load Balancers

The guest agent automatically add local routes in the following scenarios for vNICs, but not for Dynamic NICs:

• When you configure an alias IP range, the guest agent creates a local route for the alias IP range.
• When you create a target instance that references a compute instance for protocol forwarding, the guest agent creates a local route for the associated forwarding rule IP address.
• When you add a backend to a Passthrough Network Load Balancer, the guest agent creates a local route for the associated forwarding rule IP address.

Because the local routes aren't added for Dynamic NICs, the Dynamic NIC might experience dropped packets.

To resolve this issue, add the IP addresses manually as follows:

1. Connect to the instanceby using SSH.
2. If you are configuring an alias IP range, do the following. Otherwise, you can skip this step.
   1. In /etc/default/instance_configs.cfg, ensure that the ip_aliasessetting is set to true.
   2. If the ip_aliases setting is set to false, modify the file to change it to true and then restart the guest agent:
      systemctl restart google-guest-agent
3. Configure a local route for the alias IP range or the forwarding rule IP address by using the following command:
   ip route add to local IP_ADDRESS dev DYNAMIC_NIC_DEVICE_NAME proto 66
   Replace the following:
   • IP_ADDRESS: the alias IP range or forwarding rule IP address that you want to add a local route for.
   • DYNAMIC_NIC_DEVICE_NAME: the device name of the Dynamic NIC that you want to add a local route for. For example, a-gcp.ens4.3.

Issues with installation and management of Dynamic NICs in guest agent versions 20250901.00 to 20251120.01

If youconfigure automatic management of Dynamic NICsand your instance is running the guest agent at a version from 20250901.00 to 20251120.01, you might encounter the following issues:

• The guest agent fails to install and manage Dynamic NICs in the guest OS of your instance.
  You might receive an error that includes Cannot find device when running commands in the guest OS that reference Dynamic NICs.
• Deleting multiple Dynamic NICs causes themetadata server to become inaccessible.

Root cause

Starting with version 20250901.00, theguest agent migrated to a new plugin-based architecture to improve modularity. The new architecture didn't initially support the automatic installation and management of Dynamic NICs.

Resolution

To resolve these issues, update your instance to use guest agent version 20251205.00 or later:

1. To update the guest agent to the latest version, seeUpdate the guest environment.
2. To confirm the guest agent version that your instance is running, see View installed packages by operating system version.

If necessary, you can temporarily work around these issues for instances that are running guest agent versions 20250901.00 to 20251120.01 by following the instructions in Backward compatibilityto revert to the previous guest agent architecture.

What's next

• Create VMs with multiple network interfaces