Create Cloud Composer environments (original) (raw)

Skip to main content

Create Cloud Composer environments

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains how to create a Cloud Composer environment.

Before you begin

Step 1. Create or choose an environment's service account

When you create an environment, you specify a service account. This service account is called environment's service account. Your environment uses this service account to perform most of the operations.

The service account for your environment is not a user account. A service account is a special kind of account used by an application or a virtual machine (VM) instance, not a person.

You can't change the service account of your environment later.

If you don't have a service account for Cloud Composer environments in your project yet, create it.

See Create environments (Terraform) for an extended example of creating a service account for your environment in Terraform.

To create a new service account for your environment:

  1. Create a new service account as described in the Identity and Access Management documentation.
  2. Grant a role to it, as described in the Identity and Access Management documentation. The required role is Composer Worker (composer.worker).
  3. To access other resources in your Google Cloud project, grant extra permissions to access those resources to this service account. The Composer Worker (composer.worker) role provides this required set of permissions in most cases. Add extra permissions to this service account only when it's necessary for the operation of your DAGs.

Step 2. Basic setup

This step creates a Cloud Composer environment with default parameters in the specified location.

Console

  1. In the Google Cloud console, go to the Create environment page.
    Go to Create environment
  2. In the Name field, enter a name for your environment.
    The name must start with a lowercase letter followed by up to 62 lowercase letters, numbers, or hyphens, and can't end with a hyphen. The environment name is used to create subcomponents for the environment, so you must provide a name that is also valid as a Cloud Storage bucket name. See Bucket naming guidelines for a list of restrictions.
  3. In the Location drop-down list, choose a location for your environment.
    A location is the region where the environment is located.
  4. In the Image version drop-down list, select a Cloud Composer image with the required version of Airflow.
  5. In the Service account drop-down list, select a service account for your environment.
    If you don't have a service account for your environment yet, seeCreate or choose an environment's service account.

gcloud

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version IMAGE_VERSION \
    --service-account "SERVICE_ACCOUNT"

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
"

API

Construct an environments.create API request. Specify the configuration in the Environment resource.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "softwareConfig": {
      "imageVersion": "IMAGE_VERSION"
    },
    "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "softwareConfig": {
      "imageVersion": "composer-3-airflow-2.10.5-build.6"
    },
    "nodeConfig": {
      "serviceAccount": "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Terraform

To create an environment with default parameters is a specified location, add the following resource block to your Terraform configuration and runterraform apply.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    software_config {
      image_version = "IMAGE_VERSION"
    }
    node_config {
      service_account = "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {
    software_config {
      image_version = "composer-3-airflow-2.10.5-build.6"
    }
    node_config {
      service_account = "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Step 3. (Optional) Configure environment scale and performance parameters

To specify the scale and performance configuration for your environment, select the environment size and workloads configuration.

You can change all performance and scale parameters after you create an environment.

Following parameters control the scale and performance:

Console

You can select a preset for your environment. When you select a preset, the scale and performance parameters for that preset are automatically selected. You also have an option to select a custom preset and specify all scale and performance parameters for your environment.

To select the scale and performance configuration for your environment, on the Create environment page:

gcloud

When you create an environment, the following arguments control the scale and performance parameters of your environment.

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --environment-size ENVIRONMENT_SIZE \
    --scheduler-count SCHEDULER_COUNT \
    --scheduler-cpu SCHEDULER_CPU \
    --scheduler-memory SCHEDULER_MEMORY \
    --scheduler-storage SCHEDULER_STORAGE \
    --triggerer-count TRIGGERER_COUNT \
    --triggerer-cpu TRIGGERER_CPU \
    --triggerer-memory TRIGGERER_MEMORY \
    --dag-processor-cpu DAG_PROCESSOR_CPU \
    --dag-processor-memory DAG_PROCESSOR_MEMORY \
    --dag-processor-storage DAG_PROCESSOR_STORAGE \
    --web-server-cpu WEB_SERVER_CPU \
    --web-server-memory WEB_SERVER_MEMORY \
    --web-server-storage WEB_SERVER_STORAGE \
    --worker-cpu WORKER_CPU \
    --worker-memory WORKER_MEMORY \
    --worker-storage WORKER_STORAGE \
    --min-workers WORKERS_MIN \
    --max-workers WORKERS_MAX

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
    --environment-size small \
    --scheduler-count 1 \
    --scheduler-cpu 0.5 \
    --scheduler-memory 2.5GB \
    --scheduler-storage 2GB \
    --triggerer-count 1 \
    --triggerer-cpu 0.5 \
    --triggerer-memory 0.5GB \
    --dag-processor-cpu 0.5 \
    --dag-processor-memory 2GB \
    --dag-processor-storage 1GB \
    --web-server-cpu 1 \
    --web-server-memory 2.5GB \
    --web-server-storage 2GB \
    --worker-cpu 1 \
    --worker-memory 2GB \
    --worker-storage 2GB \
    --min-workers 2 \
    --max-workers 4

API

When you create an environment, in the Environment > EnvironmentConfig > WorkloadsConfig resource, specify environment scale and performance parameters.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "workloadsConfig": {
      "scheduler": {
        "cpu": SCHEDULER_CPU,
        "memoryGb": SCHEDULER_MEMORY,
        "storageGb": SCHEDULER_STORAGE,
        "count": SCHEDULER_COUNT
      },
      "triggerer": {
        "count": TRIGGERER_COUNT,
        "cpu": TRIGGERER_CPU,
        "memoryGb": TRIGGERER_MEMORY
      },
      "dagProcessor": {
        "count": 1,
        "cpu": DAG_PROCESSOR_CPU,
        "memoryGb": DAG_PROCESSOR_MEMORY,
        "storageGb": DAG_PROCESSOR_STORAGE
      },
      "webServer": {
        "cpu": WEB_SERVER_CPU,
        "memoryGb": WEB_SERVER_MEMORY,
        "storageGb": WEB_SERVER_STORAGE
      },
      "worker": {
        "cpu": WORKER_CPU,
        "memoryGb": WORKER_MEMORY,
        "storageGb": WORKER_STORAGE,
        "minCount": WORKERS_MIN,
        "maxCount": WORKERS_MAX
      }
    },
    "environmentSize": "ENVIRONMENT_SIZE",
    "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "workloadsConfig": {
      "scheduler": {
        "cpu": 2.5,
        "memoryGb": 2.5,
        "storageGb": 2,
        "count": 1
      },
      "triggerer": {
        "cpu": 0.5,
        "memoryGb": 0.5,
        "count": 1
      },
      "dagProcessor": {
        "count": 1,
        "cpu": 0.5,
        "memoryGb": 2,
        "storageGb": 1
      },
      "webServer": {
        "cpu": 1,
        "memoryGb": 2.5,
        "storageGb": 2
      },
      "worker": {
        "cpu": 1,
        "memoryGb": 2,
        "storageGb": 2,
        "minCount": 2,
        "maxCount": 4
      }
    },
    "environmentSize": "ENVIRONMENT_SIZE_SMALL",
    "nodeConfig": {
      "serviceAccount": "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Terraform

When you create an environment, following arguments control the scale and performance parameters of your environment.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {

    workloads_config {

      scheduler {
        cpu = SCHEDULER_CPU
        memory_gb = SCHEDULER_MEMORY
        storage_gb = SCHEDULER_STORAGE
        count = SCHEDULER_COUNT
      }
      triggerer {
        count = TRIGGERER_COUNT
        cpu = TRIGGERER_CPU
        memory_gb = TRIGGERER_MEMORY
      }
      web_server {
        cpu = WEB_SERVER_CPU
        memory_gb = WEB_SERVER_MEMORY
        storage_gb = WEB_SERVER_STORAGE
      }
      worker {
        cpu = WORKER_CPU
        memory_gb = WORKER_MEMORY
        storage_gb = WORKER_STORAGE
        min_count = WORKERS_MIN
        max_count = WORKERS_MAX
      }
    }

    environment_size = "ENVIRONMENT_SIZE"

    node_config {
      service_account = "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {

    workloads_config {

      scheduler {
        cpu = 2.5
        memory_gb = 2.5
        storage_gb = 2
        count = 1
      }
      triggerer {
        count = 1
        cpu = 0.5
        memory_gb = 0.5
      }
      web_server {
        cpu = 1
        memory_gb = 2.5
        storage_gb = 2
      }
      worker {
        cpu = 1
        memory_gb = 2
        storage_gb = 2
        min_count = 2
        max_count = 4
      }
    }

    environment_size = "ENVIRONMENT_SIZE_SMALL"

    node_config {
      service_account = "
example-account@example-project.iam.gserviceaccount.com
"
    }

  }
}

Step 4. (Optional) Specify a zone for the environment's database

You can specify a preferred Cloud SQL zone for your environment.

Console

On the Create environment page:

  1. In the Advanced configuration section, expand the Show advanced configuration item.
  2. In the Airflow database zone list, select a preferred Cloud SQL zone.

gcloud

When you create an environment, the --cloud-sql-preferred-zone argument specifies a preferred Cloud SQL zone.

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --cloud-sql-preferred-zone SQL_ZONE

Replace the following:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
    --cloud-sql-preferred-zone us-central1-a

API

When you create an environment, in the Environment > DatabaseConfig resource, specify the preferred Cloud SQL zone.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "databaseConfig": {
      "zone": "SQL_ZONE"
    },
      "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace the following:

Example:


// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "databaseConfig": {
      "zone": "us-central1-a"
    },
    "nodeConfig": {
      "serviceAccount": "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Terraform

When you create an environment, the zone field in the database_configblock specifies the preferred Cloud SQL zone.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    database_config {
      zone = "SQL_ZONE"
    }

    node_config {
      service_account = "SERVICE_ACCOUNT"
    }
  }
}

Replace the following:

Step 5. (Optional) Configure your environment's networking

You can configure Cloud Composer 3 networking in the following ways:

Console

  1. Make sure that your networking is configured for the type of environment that you want to create.
  2. In the Network configuration section, expand the Show network configuration item.
  3. If you want to connect your environment to a VPC network, in theNetwork attachment field, select a network attachment. You can also create a new network attachment. For more information, seeConnect an environment to a VPC network.
  4. If you want to create a Private IP environment, in the Networking typesection, select the Private IP environment option.
  5. If you want to add network tags, see Add network tags for more information.

gcloud

Make sure that your networking is configured for the type of environment that you want to create.

When you create an environment, the following arguments control the networking parameters. If you omit a parameter, the default value is used.

Example (Private IP environment with a connected VPC network)

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --enable-private-environment \
    --network NETWORK_ID \
    --subnetwork SUBNETWORK_ID \

Replace:

Network tags are applied to all node VMs in your environment's cluster. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply withRFC 1035.

For example, you might want to add network tags if you plan to restrict traffic for a Private IP environment with firewall rules.

Console

On the Create environment page:

  1. Locate the Network configuration section.
  2. In the Network tags field, enter network tags for your environment.

gcloud

When you create an environment, following arguments control network tags:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --tags TAGS

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --tags group1,production

API

When you create an environment, in the Environment > EnvironmentConfig resource, specify network tags for your environment.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "nodeConfig": {
      "tags": [
        "TAG"
      ],
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "nodeConfig": {
      "tags": [
        "group1",
        "production"
      ],
      "serviceAccount": "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Terraform

When you create an environment, following fields define network tags for your environment:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {

    node_config {
      tags = ["TAGS"]
      service_account = "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {
    node_config {
      tags = ["group1","production"]
      service_account = "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Step 7. (Optional) Configure web server network access

The Airflow web server access parameters do not depend on the type of your environment. Instead, you can configure web server access separately. For example, a Private IP environment can still have the Airflow UI accessible from the internet.

You can't configure the allowed IP ranges using private IP addresses.

Console

On the Create environment page:

  1. In the Network configuration section, expand the Show network configuration item.
  2. In the Web server network access control section:
    • To provide access to the Airflow web server from all IP addresses, select Allow access from all IP addresses.
    • To restrict access only to specific IP ranges, selectAllow access only from specific IP addresses. In the IP rangefield, specify an IP range in the CIDR notation. In the Description field, specify an optional description for this range. If you want to specify more than one range, click Add IP range.
    • To forbid access for all IP addresses, select Allow access only from specific IP addresses and click Delete item next to the empty range entry.

gcloud

When you create an environment, following arguments control web server access level:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --web-server-allow-ip ip_range=WS_IP_RANGE,description=WS_RANGE_DESCRIPTION

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
    --web-server-allow-ip ip_range=192.0.2.0/24,description="office net 1" \
    --web-server-allow-ip ip_range=192.0.4.0/24,description="office net 3"

API

When you create an environment, in the Environment > EnvironmentConfig resource, specify web server access parameters.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "webServerNetworkAccessControl": {
      "allowedIpRanges": [
        {
          "value": "WS_IP_RANGE",
          "description": "WS_RANGE_DESCRIPTION"
        }
      ]
    },
      "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:


// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "webServerNetworkAccessControl": {
      "allowedIpRanges": [
        {
          "value": "192.0.2.0/24",
          "description": "office net 1"
        },
        {
          "value": "192.0.4.0/24",
          "description": "office net 3"
        }
      ]
    },
      "nodeConfig": {
        "serviceAccount": "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Terraform

When you create an environment, the allowed_ip_range block in theweb_server_network_access_control block contains IP ranges that can access web server.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {

    web_server_network_access_control {

      allowed_ip_range {
        value = "WS_IP_RANGE"
        description = "WS_RANGE_DESCRIPTION"
      }
    }

    node_config {
      service_account = "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {

    web_server_network_access_control {
      allowed_ip_range {
        value = "192.0.2.0/24"
        description = "office net 1"
      },
      allowed_ip_range {
        value = "192.0.4.0/24"
        description = "office net 3"
      }
    }

    node_config {
      service_account = "
example-account@example-project.iam.gserviceaccount.com
"
    }

}

Step 8. (Optional) Specify Airflow configuration overrides and environment variables

You can set up Airflow configuration overrides andenvironment variables when you create an environment. As an alternative, you can do it later, after your environment is created.

Some Airflow configuration options are blocked and you can't override them.

For the list of available Airflow configuration options, seeConfiguration reference for Airflow 2and Airflow 1.10.*

To specify Airflow configuration overrides and environment variables:

Console

On the Create environment page:

  1. In the Environment variables section, clickAdd environment variable.
  2. Enter the Name and Value for the environment variable.
  3. In the Airflow configuration overrides section, clickAdd Airflow configuration override.
  4. Enter the Section, Key, and Value for the configuration option override.
    For example:
    Section Key Value
    webserver dag_orientation TB

gcloud

When you create an environment, following arguments control environment variables and Airflow configuration overrides:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --env-variables ENV_VARS \
    --airflow-configs CONFIG_OVERRIDES

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
    --env-variables SENDGRID_MAIL_FROM=user@example.com,SENDGRID_API_KEY=example-key \
    --airflow-configs core-dags_are_paused_at_creation=True,webserver-dag_orientation=TB

API

When you create an environment, in the Environment > EnvironmentConfig resource, specify environment variables and Airflow configuration overrides.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "softwareConfig": {
      "airflowConfigOverrides": {
        "SECTION-KEY": "OVERRIDE_VALUE"
      },
      "envVariables": {
        "VAR_NAME": "VAR_VALUE",
      }
    },
    "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace:

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "softwareConfig": {
      "airflowConfigOverrides": {
        "core-dags_are_paused_at_creation": "True",
        "webserver-dag_orientation": "TB"
      },
      "envVariables": {
        "SENDGRID_MAIL_FROM": "user@example.com",
        "SENDGRID_API_KEY": "example-key"
      }
    },
    "nodeConfig": {
        "serviceAccount": "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Terraform

When you create an environment, following blocks control environment variables and Airflow configuration overrides:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {

    software_config {

      airflow_config_overrides = {
        SECTION-KEY = "OVERRIDE_VALUE"
      }

      env_variables = {
        VAR_NAME = "VAR_VALUE"
      }
    }

    node_config {
      service_account = "SERVICE_ACCOUNT"
    }

  }
}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {

    software_config {

      airflow_config_overrides = {
        core-dags_are_paused_at_creation = "True"
        webserver-dag_orientation = "TB"
      }

      env_variables = {
        SENDGRID_MAIL_FROM = "user@example.com"
        SENDGRID_API_KEY = "example-key"
      }
    }

    node_config {
      service_account = "
example-account@example-project.iam.gserviceaccount.com
"
    }
  }
}

Step 9. (Optional) Specify maintenance windows

Default maintenance windows in Cloud Composer 3 are defined in the following way:

To specify custom maintenance windows for your environment:

Console

On the Create environment page

  1. Locate the Maintenance windows section.
  2. In the Timezone drop-down list, choose a time zone for maintenance windows.
  3. Set Start time, Days, and Length, so that:
    • At least 12 hours of time is allocated in a single week.
    • You can use several time slots, but each slot duration must be at least 4 hours.
      For example, a period of 4 hours every Monday, Wednesday, and Friday provides the required amount of time.

gcloud

The following arguments define maintenance windows parameters:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --maintenance-window-start 'DATETIME_START' \
    --maintenance-window-end 'DATETIME_END' \
    --maintenance-window-recurrence 'MAINTENANCE_RECURRENCE'

Replace:

The following example specifies a 6-hour maintenance window between 01:00 and 07:00 (UTC) on Wednesdays, Saturdays, and Sundays. The 1 January, 2023 date is ignored.

gcloud composer environments create example-environment \
  --location us-central1 \
  --image-version composer-3-airflow-2.10.5-build.6 \
  --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
  --maintenance-window-start '2023-01-01T01:00:00Z' \
  --maintenance-window-end '2023-01-01T07:00:00Z' \
  --maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=SU,WE,SA'

API

When you create an environment, in theEnvironment > EnvironmentConfig resource, specify maintenance windows parameters:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "config": {
    "maintenanceWindow": {
        "startTime": "DATETIME_START",
        "endTime": "DATETIME_END",
        "recurrence": "MAINTENANCE_RECURRENCE"
    },
    "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Replace:

The following example specifies a 6-hour maintenance window between 01:00 and 07:00 (UTC) on Wednesdays, Saturdays, and Sundays. The 1 January, 2023 date is ignored.

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "config": {
    "maintenanceWindow": {
        "startTime": "2023-01-01T01:00:00Z",
        "endTime": "2023-01-01T07:00:00Z",
        "recurrence": "FREQ=WEEKLY;BYDAY=SU,WE,SA"
    },
    "nodeConfig": {
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  }
}

Terraform

The maintenance_window block specifies the maintenance windows for your environment:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    maintenance_window {
      start_time = "DATETIME_START"
      end_time = "DATETIME_END"
      recurrence = "MAINTENANCE_RECURRENCE"
    }

    node_config {
      service_account = "SERVICE_ACCOUNT"
    }
  }
}

Replace:

The following example specifies a 6-hour maintenance window between 01:00 and 07:00 (UTC) on Wednesdays, Saturdays, and Sundays. The 1 January, 2023 date is ignored.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {
    maintenance_window {
      start_time = "2023-01-01T01:00:00Z"
      end_time = "2023-01-01T07:00:00Z"
      recurrence = "FREQ=WEEKLY;BYDAY=SU,WE,SA"
    }
  }
}

Step 10. (Optional) Data lineage integration

Data lineage is a Dataplex Universal Catalog feature that lets you track data movement.

Data lineage integration is available in all versions of Cloud Composer 3.

Data lineage integration is automatically enabled in a new Cloud Composer environment if the following conditions are met:

You can disable data lineage integration when you create an environment. For example, if you want to override the automatic behavior or choose toenable data lineage later, after the environment is created.

Console

To disable Data lineage integration, on the Create environment page:

  1. In the Advanced configuration section, expand the Show advanced configuration item.
  2. In the Dataplex data lineage integration section, selectDisable integration with Dataplex data lineage.

gcloud

When you create an environment, the --disable-cloud-data-lineage-integrationargument disables the data lineage integration.

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --disable-cloud-data-lineage-integration

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
    --disable-cloud-data-lineage-integration

Step 11. (Optional) Configure data encryption (CMEK)

By default, data in your environment is encrypted with a key provided by Google.

To use customer-managed encryption keys (CMEK) to encrypt data in your environment, follow the instructions outlined inUsing customer-managed encryption keys.

Step 12. (Optional) Use a custom environment's bucket

When you create an environment, Cloud Composer creates a bucket for your environment automatically.

As an alternative, you can specify a custom Cloud Storage bucket from your project. Your environment uses this bucket in the same way as the automatically created bucket.

To use a custom environment bucket, follow the instructions outlined inUse a custom environment's bucket.

Step 13. (Optional) Configure database retention

If you enable database retention in your environment, then Cloud Composer periodically removes records related to DAG executions and user sessions older than the specified time period from the Airflow database. The most recent DAG run information is always retained.

By default, database retention is enabled. To configure the retention period for a new environment or disable database retention, follow the instructions outlined inConfigure database retention policy. You can also configure database retention later.

Step 14. (Optional) Specify environment labels

You can assign labels to your environments tobreak down billing costs based on these labels.

Console

On the Create environment page, in the Labels section:

  1. Click Add label.
  2. In Key and Value fields, specify key and value pairs for the environment labels.

gcloud

When you create an environment, the --labels argument specifies a comma-separated list of keys and values with environment labels.

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "SERVICE_ACCOUNT" \
    --labels LABELS

Replace:

Example:

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-3-airflow-2.10.5-build.6 \
    --service-account "
example-account@example-project.iam.gserviceaccount.com
" \
    --labels owner=engineering-team,env=production

API

When you create an environment, in the Environmentresource, specify labels for your environment.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "labels": {
    "LABEL_KEY": "LABEL_VALUE"
  }
}

Replace:

Example:


// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "labels": {
    "owner": "engineering-team",
    "env": "production"
  }
}

Terraform

When you create an environment, specify labels in the labels block ( outside of the config block).

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  labels = {
    LABEL_KEY = "LABEL_VALUE"
  }

}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  labels = {
    owner = "engineering-team"
    env = "production"
  }

}

What's next

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-06-13 UTC.