Configure encryption with customer-managed encryption keys (original) (raw)

Managed Airflow (Gen 3) | Managed Airflow (Gen 2) | Managed Airflow (Legacy Gen 1)

This page describes how to useCustomer Managed Encryption Keys (CMEK) to protect Managed Airflow environments.

By default, Managed Service for Apache Airflow encrypts customer content at rest. Managed Airflow handles encryption for you without any additional actions on your part. This option is called Google default encryption.

If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in Cloud KMS with CMEK-integrated services including Managed Airflow. Using Cloud KMS keys gives you control over their protection level, location, rotation schedule, usage and access permissions, and cryptographic boundaries. Using Cloud KMS also lets you view audit logs and control key lifecycles. Instead of Google owning and managing the symmetrickey encryption keys (KEKs) that protect your data, you control and manage these keys in Cloud KMS.

After you set up your resources with CMEKs, the experience of accessing your Managed Airflow resources is similar to using Google default encryption. For more information about your encryption options, see Customer-managed encryption keys (CMEK).

Data protected with CMEK encryption

Managed Airflow protects the following data with CMEK encryption:

• Contents and schema of the Airflow database
• Environment and Airflow task logs in Cloud Logging
• Contents of the environment's bucket
• Secrets stored in the environment's cluster
• Persistent disks used by the task queue
• Container images of environment components stored in Artifact Registry repositories

For specific details about how the data is encrypted, see:

• About customer-managed encryption keys (CMEK) in the Cloud SQL documentation.
• Use customer-managed encryption keys (CMEK) andEncrypt secrets at the application layer in the Google Kubernetes Engine documentation.
• Customer-managed encryption keys in the Cloud Storage documentation.
• Configure CMEK for Cloud Logging in the Cloud Logging documentation.

Data not protected with CMEK encryption

Cloud Monitoring does not support CMEK encryption. The name of your environment and names of DAGs are stored in the Monitoring database in the encrypted form using Google-owned and Google-managed encryption keys.

Managed Airflow stores the following information protected with Google-owned and Google-managed encryption keys, not customer-managed keys:

• Environment name
• Airflow configuration overrides
• Environment variables
• Labels
• The names of some parameters stored by Managed Airflow can include a sub-string of environment's name.

Rotation of CMEK keys for Managed Airflow

After you configure encryption in your environment using CMEK keys, you might also want to consider rotating these keys on a regular basis as described in KMS documentation.

When you rotate a CMEK key, data encrypted with previous key versions isnot automatically re-encrypted with the new key version. For more information, see Re-encrypting data.

Specifically, this applies to:

• Objects stored in the environment's bucket.
• Data stored in the Airflow database.
• All other data objects encrypted with CMEK in the Managed Airflow environment.
• Container images of environment components stored in Artifact Registry repositories.

Use a customer-managed encryption key for your environment

Before you begin

• You can configure CMEK only when you create an environment. It is not possible to enable CMEK for an existing environment.
• Managed Airflow supports CMEK encryption usingkeys stored in External Key Managers.
• You must create a CMEK key in the same region where your environments are located. You cannot use multi-regional or global keys.
• If you want your environment to run inside aVPC Service Controls perimeter, you must add the Cloud Key Management Service API to the perimeter.

Step 1. Create a customer-managed encryption key

Follow the steps outlined inCreating symmetric encryption keys to create a key in the region where your environment is located.

Step 2. Grant roles to service agents

Console

Skip this step. You grant permissions toservice agents when youspecify a key for your environment.

gcloud

The following service agents must have the Cloud KMS CryptoKey Encrypter/Decrypter role on the key that you use for your environment.

Replace PROJECT_NUMBER with yourproject's number.

1. (If required) If some of these service accounts are not present in your project, it means that an identity for this service is not yet created. This can happen, for example, if you did not yet create any Managed Airflow environments in your project.
   To add these service accounts, create identities for the listed services with the following command:

gcloud beta services identity create \  
  --service=API_SERVICE_NAME  

Replace API_SERVICE_NAME with theAPI service name of a service that does not have a service account in your project.
Example:

gcloud beta services identity create \  
  --service=composer.googleapis.com  

1. Grant permissions to service agents:
   1. Grant the role to Managed Airflow Service Agent:

   gcloud kms keys add-iam-policy-binding KEY_NAME \  
     --location KEY_LOCATION \  
     --keyring KEY_RING_NAME \  
     --member=serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com \  
     --role=roles/cloudkms.cryptoKeyEncrypterDecrypter \  
     --project KEY_PROJECT_ID  

   1. Grant encrypt/decrypt permissions to Cloud Storage Service Agent. You can skip this step if you use a custom environment's bucket.

   gcloud storage service-agent \  
     --authorize-cmek=projects/KEY_PROJECT_ID/locations/KEY_LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME  

Replace:

• PROJECT_ID with yourproject's ID.
• KEY_PROJECT_ID with the ID of the project that stores your customer-managed key. If you use a key from another project, this value is different from your project's ID. If you use a key from the same project, this value is your project's ID.
• PROJECT_NUMBER with yourproject's number.
• KEY_LOCATION with your customer-managed key location. This location must be the same as your environment's location.
• KEY_NAME with your customer-managed key name.
• KEY_RING_NAME with the keyring that stores your customer-managed key.
  To get these values, you can run gcloud projects describe,gcloud kms keyrings list, and gcloud kms keys describe commands.

Step 3. Create an environment with CMEK

After you create a customer-managed encryption key, you can use it to create Managed Airflow environments.

Console

When you create an environment:

1. Expand the Advanced configuration section. In the Data encryption section, selectCloud Key Management Service key.
2. In the Select a Cloud Key Management Service key drop-down list, select your key.
3. If additional setup is required, a message appears to inform you. In this case:
   1. Click Open wizard.
   2. In the Prepare CMEK key for usage in Managed Airflowdialog, view the list of service agentsthat must have the Cloud KMS CryptoKey Encrypter/Decrypter role on the key.
   3. To grant the required roles and permissions, click Grant.

gcloud

The --kms-key argument specifies a customer-managed enryption key for your environment.

For more information about creating environments, seeCreate environments. For example, you might want to specify other parameters for your environment.

gcloud composer environments create ENVIRONMENT_NAME \
  --location LOCATION \
  --image-version IMAGE_VERSION \
  --kms-key projects/KEY_PROJECT_ID/locations/KEY_LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• IMAGE_VERSION with the name of the Managed Airflow image.
• KEY_PROJECT_ID with the ID of the project where the key is located. If you use a key from another project, this value is different from your project's ID. If you use a key from the same project, this value is your project's ID.
• LOCATION with the region where the environment is located.
• KEY_LOCATION with the location of your customer-managed key. This location must be the same as your environment's location.
• KEY_NAME with your customer-managed key name.
• KEY_RING_NAME with the keyring that stores your customer-managed key.

Example:

gcloud composer environments create example-environment \
  --location us-central1 \
  --image-version composer-3-airflow-2.11.1-build.6 \
  --kms-key projects/example-project/locations/us-central1/keyRings/example-key-ring/cryptoKeys/example-key

View the environment's encryption configuration

You can view encryption configuration for an existing environment:

Console

1. In Google Cloud console, go to the Environments page.
   Go to Environments
2. In the list of environments, click the name of your environment. The Environment details page opens.
3. Go to the Environment configuration tab.
4. Details about encryption are listed in the Data encryption key item.

gcloud

Run the following gcloud command to view the encryption configuration

gcloud composer environments describe \
  ENVIRONMENT_NAME \
  --location LOCATION \
  --format="value(config.encryptionConfig)"

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.

Example:

gcloud composer environments describe \
  example-environment \
  --location us-central1 \
  --format="value(config.encryptionConfig)"

Use CMEK for Managed Airflow logs

Cloud Logging supports encrypting Log storage with CMEK keys. We recommend to use the standard CMEK procedure for encrypting logs with CMEK keys.

To encrypt logs with CMEK keys, follow the instructions outlined inManage the keys that protect Logging storage data.

Redirect Managed Airflow logs to a CMEK-encrypted Cloud Storage bucket

If you expect your logs to contain sensitive data, you might want to redirect Managed Airflow logs to a CMEK-encrypted Cloud Storage bucket by usingLog Router. Doing so prevents your logs from being sent to Monitoring.

If you require support from Cloud Customer Care, you might need to grant Google support engineers access to the Managed Airflow logs stored in Cloud Storage.

gcloud

1. Create a new Cloud Storage bucket to store the logs.

gcloud storage buckets create gs://BUCKET_NAME --location=LOCATION  

Replace:

• LOCATION with the region where the environment is located.
• BUCKET_NAME with the name for the bucket.
  Example:

gcloud storage buckets create gs://composer-logs-us-central1-example-environment --location=us-central1  

1. Encrypt the bucket with your CMEK key.

gcloud storage buckets update gs://BUCKET_NAME \  
  --default-encryption-key=projects/KEY_PROJECT_ID/locations/KEY_LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME  

Replace:

• KEY_PROJECT_ID with the ID of the project where the key is located. If you use a key from another project, this value is different from your project's ID. If you use a key from the same project, this value is your project's ID.
• KEY_LOCATION with the location of your customer-managed key. This location must be the same as your environment's location.
• KEY_RING_NAME with the keyring that stores your customer-managed key.
• KEY_NAME with your customer-managed key name.
• BUCKET_NAME with the name of the bucket.
  Example:

gcloud storage buckets update gs://composer-logs-us-central1-example-environment \  
  --default-encryption-key=projects/example-project/locations/us-central1/keyRings/example-key-ring/cryptoKeys/example-key  

1. Create a new log sink.

gcloud logging sinks create \  
composer-log-sink-ENVIRONMENT_NAME \  
storage.googleapis.com/BUCKET_NAME \  
--log-filter "resource.type=cloud_composer_environment AND resource.labels.environment_name=ENVIRONMENT_NAME AND resource.labels.location=LOCATION"  

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.
• BUCKET_NAME with the name of the bucket.
  Example:

gcloud logging sinks create \  
composer-log-sink-example-environment \  
storage.googleapis.com/composer-logs-us-central1-example-environment \  
--log-filter "resource.type=cloud_composer_environment AND resource.labels.environment_name=example-environment AND resource.labels.location=us-central1"  

1. Grant the Storage Object Creator role to the service account for this bucket. The service account is shown in the result of the previous command).

gcloud projects add-iam-policy-binding \  
  PROJECT_ID \  
  --member="serviceAccount:LOGGING_SERVICE_AGENT" \  
  --role="roles/storage.objectCreator" \  
  --condition=None  

Replace:

• PROJECT_ID with the Project ID.
• LOGGING_SERVICE_AGENT with the Logging service agent account for this bucket. The name of this account is obtained on the previous step.
  Example:

gcloud projects add-iam-policy-binding \  
  example-project \  
  --member="serviceAccount:example-sa@gcp-sa-logging.iam.gserviceaccount.com" \  
  --role="roles/storage.objectCreator" \  
  --condition=None  

1. Exclude the logs for your new environment from Monitoring.

gcloud beta logging sinks update _Default \  
  --add-exclusion name=ENVIRONMENT_NAME-exclusion,filter="resource.type=cloud_composer_environment AND resource.labels.environment_name=ENVIRONMENT_NAME AND resource.labels.location=LOCATION"  

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.
  Example:

gcloud beta logging sinks update _Default \  
  --add-exclusion name=example-environment-exclusion,filter="resource.type=cloud_composer_environment AND resource.labels.environment_name=example-environment AND resource.labels.location=us-central1"  

1. Add organization-level CMEK encryption to the Log Router.

gcloud logging cmek-settings describe \  
  --organization=ORGANIZATION_ID  

gcloud kms keys add-iam-policy-binding \  
      --project=KEY_PROJECT_ID \  
      --member LOGGING_SERVICE_AGENT \  
      --role roles/cloudkms.cryptoKeyEncrypterDecrypter \  
      --location=KEY_LOCATION \  
      --keyring=KEY_RING_NAME \  
      KEY_NAME  

gcloud logging cmek-settings update \  
  --organization=ORGANIZATION_ID \  
  --kms-project=KEY_PROJECT_ID \  
  --kms-keyring=KEY_RING_NAME \  
  --kms-location=KEY_LOCATION \  
  --kms-key-name=KEY_NAME  

Replace:

• ORGANIZATION_ID with yourorganization ID.
• KEY_PROJECT_ID with the ID of the project where the key is located. If you use a key from another project, this value is different from your project's ID. If you use a key from the same project, this value is your project's ID.
• KEY_RING_NAME with the keyring that stores your customer-managed key.
• KEY_LOCATION with the location of your customer-managed key. This location must be the same as your environment's location.
• KEY_NAME with your customer-managed key name.

What's next

• Security overview
• Create environments