🚨 Action Required: Ensure that you no longer use gcr.io/kubebuilder images · kubernetes-sigs/kubebuilder · Discussion #3907 (original) (raw)

⚠️ IMPORTANT ⚠️

If your project uses gcr.io/kubebuilder/kube-rbac-proxy it will be affected. Your project may fail to work if the image cannot be pulled. You must move as soon as possible, sometime from early 2025, the GCR will go away.

Unfortunately, we're unable to provide any guarantees regarding timelines or potential extensions at this time. Images provided under GRC will be unavailable from March 18, 2025, as per announcement. However, gcr.io/kubebuilder/ may be unavailable before this date due to efforts to deprecate infrastructure.

Key Update

kube-rbac-proxy was historically used to protect the metrics endpoint. However, its usage has been discontinued in Kubebuilder.
The default scaffold now leverages the WithAuthenticationAndAuthorization feature provided by Controller-Runtime.

This feature provides integrated support for securing metrics endpoints by embedding authentication (authn) and authorization (authz) mechanisms directly into the controller manager's metrics server, replacing the need for (https://github.com/brancz/kube-rbac-proxy) to secure metrics endpoints.

Why This Matters

What You Need to Do

Follow some options.

Option 1 - Upgrade the Project to the Latest Release (Recommended)

Upgrade your project to the latest release by re-scaffolding it and reintegrating your custom code.

Option 2 - Manually Modify Your Project

If you prefer not to fully upgrade, modify your project to use the built-in authn/authz protection via Controller-Runtime.

Option 3 - Continue Using the kube-rbac-proxy Image (Not adopt or promoted by Kubebuilder)

If you want to continue using kube-rbac-proxy, source the image from an alternative location, at your own risk. Examples include:

To mitigate risks, we manually mirror the images to registry.k8s.io/kubebuilder/kube-rbac-proxy. This registry is managed by the #sig-k8s-infra. However, we cannot promote the kube-rbac-proxy images on this registry or recommend their usage since they have been discontinued from the project.

FAQ

Why does the metrics endpoint need to be protected?

Unprotected metrics endpoints can expose sensitive data, such as system performance and application behavior, to unauthorized users. This can lead to security vulnerabilities where attackers gain insights into the system's operation and exploit weaknesses.

How can the metrics endpoint be protected?

The following are some options with details and info for those who were introduced to support and helpers in Kubebuilder.

  1. (Protection enabled by default from release v4.1.0)
    Use Controller-Runtime's feature WithAuthenticationAndAuthorization to enable authn/authz for metrics endpoints. For reference, see the code from the release 4.3.1
  2. (Optional helper introduced from release v4.2.0)
    Use NetworkPolicies to secure metrics endpoints.
    Example configuration: NetworkPolicy Example
    Also, feel free to check the (external code example)
  3. Integrate cert-manager:
    Secure the metrics endpoint using TLS encryption with cert-manager. For example, ensure that you use valid certficates such as:

Why doesn't Network Policy provide the same level of protection as kube-rbac-proxy?

NetworkPolicy acts as a firewall for pods, controlling traffic flow at the IP or port level. However, it doesn’t handle authentication (authn), authorization (authz), or encryption like kube-rbac-proxy does.

How can I manually change my project to switch to Controller-Runtime's built-in auth protection?

Following these steps should help you resolve the issue. However, there may be a few caveats depending on how old the version used to create the project is, especially if it has never been upgraded to the latest versions by re-creating the project and adding your code changes on top.

  1. Remove kube-rbac-proxy configurations:
    Ensure that you remove the container with the name: kube-rbac-proxy. Example from an old version
  2. Ensure that your manager will have the args for the metrics service and binding at the same port
  1. Enable built-in auth in main.go:
// if the enable-http2 flag is false (the default), http/2 should be disabled
// due to its vulnerabilities. More specifically, disabling http/2 will
// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
// Rapid Reset CVEs. For more information see:
// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
// - https://github.com/advisories/GHSA-4374-p667-p6c8
disableHTTP2 := func(c *tls.Config) {
setupLog.Info("disabling http/2")
c.NextProtos = []string{"http/1.1"}
}
if !enableHTTP2 {
tlsOpts = append(tlsOpts, disableHTTP2)
}
webhookServer := webhook.NewServer(webhook.Options{
TLSOpts: tlsOpts,
})
// Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
// More info:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/server
// - https://book.kubebuilder.io/reference/metrics.html
metricsServerOptions := metricsserver.Options{
BindAddress: metricsAddr,
SecureServing: secureMetrics,
TLSOpts: tlsOpts,
}
if secureMetrics {
// FilterProvider is used to protect the metrics endpoint with authn/authz.
// These configurations ensure that only authorized users and service accounts
// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
// https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/filters#WithAuthenticationAndAuthorization
metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
// TODO(user): If CertDir, CertName, and KeyName are not specified, controller-runtime will automatically
// generate self-signed certificates for the metrics server. While convenient for development and testing,
// this setup is not recommended for production.
}
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
Metrics: metricsServerOptions,
WebhookServer: webhookServer,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "da1d9c86.testproject.org",
// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
// when the Manager ends. This requires the binary to immediately end when the
// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
// speeds up voluntary leader transitions as the new leader don't have to wait
// LeaseDuration time first.
//
// In the default scaffold provided, the program ends immediately after
// the manager stops, so would be fine to enable this option. However,
// if you are doing or is intended to do any operation such as perform cleanups
// after the manager stops then its usage might be unsafe.
// LeaderElectionReleaseOnCancel: true,
})
if err != nil {
setupLog.Error(err, "unable to start manager")
os.Exit(1)
}

Note: Please ensure that you disable HTTP/2 by default, as you see in the above example. Disable HTTP/2 still required: kubernetes/kubernetes#121197

5 - Add e2e tests to validate the metrics endpoint.

Projects created with the latest versions are scaffolded with comprehensive E2E tests, including code to validate the metrics endpoint. Example. See: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/test/e2e/e2e_test.go#L166-L235

Note that you can remove the Prometheus block if you are not providing this integration.

  1. Build and Test Manually:

Build your project and ensure the metrics endpoint is working and protected with RBAC as expected. The RBAC permissions scaffolded under config/rbac should provide the required permissions. However, if you face issues, you might want to look at the last scaffolds to generate them properly. See that under testdata; we have examples.

Steps to Verify Metrics with curl manually

  1. Create the role-binding

kubectl create clusterrolebinding -metrics-binding
--clusterrole=-metrics-reader
--serviceaccount=-system:-controller-manager

  1. Generate the token to authenticate in the metrics

export TOKEN=$(kubectl create token operator-controller-controller-manager -n olmv1-system) echo $TOKEN

  1. Run the curl pod to perform the tests

kubectl run curl-metrics --rm -it --restart=Never
--image=curlimages/curl:7.87.0 -n -system -- /bin/sh

  1. Call the metrics enpoint using the TOKEN

curl -v -k -H "Authorization: Bearer $TOKEN" https://-controller-manager-metrics-service.-system.svc.cluster.local:8443/metrics

❓ Why is this happening?

The kube-rbac-proxy images have been rebuilt and re-tagged by Kubebuilder for an extended period. However, due to infrastructure changes within the Kubernetes ecosystem and the deprecation of Google Cloud Platform’s Container Registry (details here), continuing to maintain these images is no longer feasible.

Additionally, the project has been in the process of being donated to SIG-Auth for an extended period. Despite these efforts, significant requirements still need to be addressed before SIG-Auth can consider adopting the project. The latest review outlined several tasks being tracked here: kube-rbac-proxy issue #238.

For more details, refer to:

Why can't we recommend just replacing the kube-rbac-proxy image with the latest version from another trusted source?

The following are some key reasons.

Please update your configurations accordingly to avoid disruptions. If you have any questions or need further assistance, feel free to ask in this discussion thread or the Kubebuilder Slack channel.

For further information, check the metrics section in the documentation: https://book.kubebuilder.io/reference/metrics.