Using GKE Dataplane V2 (original) (raw)

This page explains how to enable and troubleshootGKE Dataplane V2 for Google Kubernetes Engine (GKE) clusters.

New Autopilot clusters have GKE Dataplane V2 enabled in versions 1.22.7-gke.1500 and later and versions 1.23.4-gke.1500 and later. If you're experiencing issues with using GKE Dataplane V2, skip toTroubleshooting.

Creating a GKE cluster with GKE Dataplane V2

You can enable GKE Dataplane V2 when you create new clusters with GKE version 1.20.6-gke.700 and later by using the gcloud CLI or the GKE API. You can also enable GKE Dataplane V2 inPreview when you create new clusters with GKE version 1.17.9 and later

Console

To create a new cluster with GKE Dataplane V2, perform the following tasks:

  1. In the Google Cloud console, go to the Create a Kubernetes cluster page.
    Go to Create a Kubernetes cluster
  2. In the Networking section, select the Enable Dataplane V2 checkbox. The Enable Kubernetes Network Policy option is disabled when you select Enable Dataplane V2 because network policy enforcement is built into GKE Dataplane V2.
  3. Click Create.

gcloud

To create a new cluster with GKE Dataplane V2, use the following command:

gcloud container clusters create CLUSTER_NAME \
    --enable-dataplane-v2 \
    --enable-ip-alias \
    --release-channel CHANNEL_NAME \
    --location COMPUTE_LOCATION

Replace the following:

API

To create a new cluster with GKE Dataplane V2, specify thedatapathProvider fieldin thenetworkConfig objectin your clustercreate request.

The following JSON snippet shows the configuration needed to enable GKE Dataplane V2:

"cluster":{
   "initialClusterVersion":"VERSION",
   "ipAllocationPolicy":{
      "useIpAliases":true
   },
   "networkConfig":{
      "datapathProvider":"ADVANCED_DATAPATH"
   },
   "releaseChannel":{
      "channel":"CHANNEL_NAME"
   }
}

Replace the following:

Troubleshooting issues with GKE Dataplane V2

This section shows you how to investigate and resolve issues with GKE Dataplane V2.

  1. Confirm that GKE Dataplane V2 is enabled:
kubectl -n kube-system get pods -l k8s-app=cilium -o wide

If GKE Dataplane V2 is running, the output includes Pods with the prefixanetd-. anetd is the networking controller for GKE Dataplane V2. 2. If the issue is with services or network policy enforcement, check theanetd Pod logs. Use the following log selectors in Cloud Logging: 

resource.type="k8s_container"  
labels."k8s-pod/k8s-app"="cilium"  
resource.labels.cluster_name="CLUSTER_NAME"
  1. If Pod creation is failing, check the kubelet logs for clues. Use the following log selectors in Cloud Logging:
resource.type="k8s_node"  
log_name=~".*/logs/kubelet"  
resource.labels.cluster_name="CLUSTER_NAME"

Replace CLUSTER_NAME with the name of the cluster, or remove it entirely to see logs for all clusters. 4. If the anetd Pods are not running, examine the cilium-config ConfigMap for any modifications. Avoid altering existing fields within this ConfigMap, because such changes can destabilize the cluster and disrupt anetd. The ConfigMap gets patched back to the default state only if new fields are added to it. Any changes to existing fields are not patched back, and we recommend not changing or customizing the ConfigMap.

Known issues

When you use GKE Dataplane V2, you might encounter the following known issues.

Connection timeouts for not-ready Pods

When a Pod is not ready, connections to the associated Service can time out. This is the expected behavior for GKE Dataplane V2, and it differs from kube-proxy, which can return a faster connection refused error.

Identity-Relevant Label filtering for Cilium Identity doesn't take effect and Pods are stuck in ContainerCreating state

Affected versions: 1.34, 1.35

In GKE Dataplane V2 clusters, emergency use of Identity-Relevant Label filtering via kube-system/cilium-config-emergency-override ConfigMap is not correctly applied at the affected versions.

This approach limits which Pod labels are used for Cilium Identity generation.

When other mechanisms of preventing/removing high cardinality label key/values from Pods is not available (such as when labels are applied by a tool or framework), Identity-Relevant Label filtering can be used to exclude the label keys from Cilium Identity calculation. For more information about configuring these rules, seeIdentity-Relevant Labelsin the Cilium documentation.

For the affected GKE versions, Cilium identities created by the operator continue to include the excluded labels.

Symptoms

  {"level":"warning", "msg":"Error changing endpoint identity", "error":"unable to resolve identity: timed out waiting for cilium-operator to allocate CiliumIdentity for key ...;, error: exponential backoff cancelled via context: context canceled", "k8sPodName":"...", "subsys":"endpoint"}

Fixed versions

To fix this issue,upgrade your clusterto one of the following GKE versions:

Workaround

As a workaround, apply the label filtering rules to the data.labels field in the main cilium-config ConfigMap and remove them fromcilium-config-emergency-override. This situation persists through control plane operations, such as upgrades, because GKE preserves user modifications to fields it does not manage within the cilium-config ConfigMap.

  1. Remove the labels key from the data section of thecilium-config-emergency-override ConfigMap if it exists.
  2. Edit the cilium-config ConfigMap by adding or modifying the labels key in the data section. For example, to prevent labels named uuid from being used for identity generation:
apiVersion: v1  
kind: ConfigMap  
metadata:  
  name: cilium-config  
  namespace: kube-system  
data:  
  # ... other existing keys  
  labels: "!uuid"  
  # ... other existing keys
  1. Restart the anet-operator on the control plane by upgrading the control plane to the same version it is running. This forces the operator to restart and reload its configuration:
gcloud container clusters upgrade CLUSTER_NAME \  
    --location CLUSTER_LOCATION \  
    --project PROJECT_ID \  
    --cluster-version $(gcloud container clusters describe CLUSTER_NAME --location CLUSTER_LOCATION --project PROJECT_ID --format="value(currentMasterVersion)") \  
    --master
  1. After the control plane restarts, restart the anetd DaemonSet to ensure node agents also pick up any required changes:
kubectl rollout restart daemonset anetd -n kube-system

In GKE Dataplane V2 clusters, intermittent connectivity problems can occur for masqueraded traffic or with ephemeral port usage. These problems are due to the potential port conflicts with the reserved NodePort range and typically happen in the following scenarios:

Review your ip-masq-agent configuration and ephemeral port range settings to ensure they don't conflict with the NodePort range. If you encounter intermittent connectivity issues, consider these potential causes and adjust your configuration accordingly.

Connectivity issues with hostPort in GKE Dataplane V2 clusters

Affected GKE versions: 1.29 and later

In clusters that use GKE Dataplane V2, you might encounter connectivity failures when traffic targets a node's IP:Port where port is the hostPort defined on the Pod. These issues arise in two primary scenarios:

ports:  
- containerPort: 62000  
  hostPort: 62000  
  protocol: TCP  
  hostIP: 35.232.62.64  
- containerPort: 60000  
  hostPort: 60000  
  protocol: TCP  
  hostIP: 35.232.62.64  
  # Assuming 35.232.62.64 is the external IP address of a passthrough Network Load Balancer.

We don't plan to fix these limitations in later versions. The root cause of these issues is related to Cilium's behavior and outside the direct control of GKE.

Recommendation: We recommend that you migrate to NodePort Services instead ofhostPort for improved reliability. NodePort Services provide similar capabilities.

Network Policy port ranges don't take effect

If you specify an endPort field in a Network Policy on a cluster that has GKE Dataplane V2 enabled, it won't take effect.

Kubernetes Network Policy APIlets you specify a range of ports where the Network Policy is enforced. This API is supported in clusters with Calico Network Policy but is not supported in clusters with GKE Dataplane V2.

You can verify the behavior of your NetworkPolicy objects by reading them back after writing them to the API server. If the object still contains theendPort field, the feature is enforced. If the endPort field is missing, the feature is not enforced. In all cases, the object stored in the API server is the source of truth for the Network Policy.

For more information seeKEP-2079: Network Policy to support Port Ranges.

Fixed versions

To fix this issue, upgrade your cluster to GKE versions 1.32 or later

Network Policy drops a connection due to incorrect connection tracking lookup

When a client Pod connects to itself using a Service or the virtual IP address of an internal passthrough Network Load Balancer, the reply packet is not identified as a part of an existing connection due to incorrect conntrack lookup in the dataplane. This means that a Network Policy that restricts ingress traffic for the Pod is incorrectly enforced on the packet.

The impact of this issue depends on the number of configured Pods for the Service. For example, if the Service has 1 backend Pod, the connection always fails. If the Service has 2 backend Pods, the connection fails 50% of the time.

Fixed versions

To fix this issue, upgrade your cluster to one of the following GKE versions:

Workarounds

You can mitigate this issue by configuring the port and containerPortin the Service manifest to be the same value.

Packet drops for hairpin connection flows

When a Pod creates a TCP connection to itself using a Service, such that the Pod is both the source and destination of the connection, GKE Dataplane V2 eBPF connection tracking incorrectly tracks the connection states, leading to leaked conntrack entries.

When a connection tuple (protocol, source/destination IP, and source/destination port) has been leaked, new connections using the same connection tuple might result in return packets being dropped.

Fixed versions

To fix this issue, upgrade your cluster to one of the following GKE versions:

Workarounds

Use one of the following workarounds:

Upgrade of GKE control plane causes anetd Pod deadlock

When you upgrade a GKE cluster that has GKE Dataplane V2 (advanced datapath) enabled from version 1.27 to 1.28, you might encounter a deadlock situation. Workloads might experience disruptions due to the inability to terminate old Pods or schedule necessary components like anetd.

Cause

The cluster upgrade process increases the resource requirement for the GKE Dataplane V2 components. This increase might lead to resource contention, which disrupts communication between the Cilium Container Network Interface (CNI) plugin and the Cilium daemon.

Symptoms

You might see the following symptoms:

1rpc error: code = Unknown desc = failed to destroy network for sandbox "[sandbox_id]": plugin type="cilium-cni" failed (delete): unable to connect to Cilium daemon... connection refused

Workaround

Standard clusters: To resolve the issue and allow the anetd Pod to be scheduled, temporarily increase the allocatable resources on the affected node.

  1. To identify the affected node and to check its allocatable CPU and memory, run the following command:
kubectl get nodes $NODE_NAME -o json | jq '.status.allocatable | {cpu, memory}'
  1. To temporarily increase the allocatable CPU and memory, run the following command:
kubectl patch node $NODE_NAME -p '{"status":{"allocatable":{"cpu":CPU_VALUE, "memory":MEMORY_VALUE}}}'

Autopilot clusters: To resolve the deadlock issue on Autopilot clusters, free up resources by force deleting the affected Pod:

kubectl delete pod POD_NAME -n NAMESPACE --grace-period=0 --force

Replace the following:

After you increase the allocatable resources on the node and when the upgrade from GKE version 1.27 to 1.28 completes, the anetd Pod runs on the newer version.

Nodes in NodeNotReady state due to missing containerID error

When clusters are upgraded to GKE version 1.35.1-gke.1616000 and later, nodes might immediately enter a NodeNotReady state if both GKE Dataplane V2 and Cloud Service Mesh are enabled.

Cause

Starting with GKE version 1.35.1-gke.1616000, GKE Dataplane V2 clusters use CNI version 1.1.0 in their CNI configuration files. This change requires downstream CNI plugins, such as Google Managed Istio, to also support CNI version 1.1.0. Because of a delay in the Managed Istio rollout, some clusters have not yet received the compatible version (1.23), leading to the initialization failure.

Symptoms

Affected nodes immediately show as NodeNotReady. The following error message appears in the containerd logs:

NetworkPluginNotReady message:Network plugin returns error: missing containerID

Workaround

To mitigate the issue, downgrade the impacted cluster to a GKE version earlier than 1.35.1-gke.1616000.

Custom eBPF programs interference

GKE uses eBPF programs to manage networking for GKE Dataplane V2. If you deploy custom eBPF programs on GKE-managed node network interfaces, these programs can interfere with GKE-managed eBPF programs and cause networking issues.

GKE doesn't support custom eBPF programs attached to the following network interfaces:

The presence of custom eBPF programs on these interfaces can interfere with the GKE Dataplane V2 anetd agent-installed programs, which can disrupt cluster networking. We recommend that you remove any custom eBPF programs or workloads that inject such programs from your cluster.

Discover custom eBPF programs

To discover custom eBPF programs running on cluster nodes, you can create a DaemonSet configured with the hostNetwork: true setting, that usesbpftoolto query such eBPF programs:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bpftool-logger
  labels:
    app: bpftool-logger
spec:
  selector:
    matchLabels:
      app: bpftool-logger
  template:
    metadata:
      labels:
        app: bpftool-logger
    spec:
      hostPID: true
      hostNetwork: true
      containers:
      - name: bpftool
        image: ubuntu:22.04
        securityContext:
          privileged: true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        command:
        - /bin/bash
        - -c
        - |
          echo "Installing dependencies..."
          apt-get update -y > /dev/null 2>&1
          apt-get install -y curl tar > /dev/null 2>&1

          echo "Downloading and setting up bpftool..."
          curl -sL https://github.com/libbpf/bpftool/releases/download/v7.7.0/bpftool-v7.7.0-amd64.tar.gz | tar xz
          chmod +x bpftool
          mv bpftool /usr/local/bin/

          echo "========== <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>d</mi><mi>a</mi><mi>t</mi><mi>e</mi><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi><mi>N</mi><mi>o</mi><mi>d</mi><mi>e</mi><mo>:</mo></mrow><annotation encoding="application/x-tex">(date) | Node: </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">d</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span><span class="mclose">)</span><span class="mord">∣</span><span class="mord mathnormal" style="margin-right:0.10903em;">N</span><span class="mord mathnormal">o</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span></span></span></span>{NODE_NAME} =========="
          bpftool net | grep -E '^(eth|ens4|lo|cilium|gke|veth)' | grep -v ' cil_'
          sleep infinity
  1. Save the manifest as ebpf-discovery.yaml and apply the DaemonSet:
kubectl apply -f ebpf-discovery.yaml
  1. Wait for the Pods to be running:
kubectl rollout status ds/bpftool-logger
  1. Check the logs from the Pods to discover eBPF programs:
kubectl logs -l app=bpftool-logger
  1. When you have finished, delete the DaemonSet:
kubectl delete -f ebpf-discovery.yaml

What's next