Upgrade environments (original) (raw)

Managed Airflow (Gen 3) | Managed Airflow (Gen 2) | Managed Airflow (Legacy Gen 1)

This page describes how to upgrade your environment to a new Managed Airflow or Airflow version.

About upgrade operations

You can upgrade the Managed Airflow version, the Airflow version, or both at the same time.

When you change the version of Airflow or Managed Airflow used by your environment:

Upgrading doesn't change how you connect to the resources in your environment, such as the URL of your environment's bucket, or Airflow web server.

Limitations of upgrade operations

Upgrade operations have the following limitations:

Before you begin

Check that your environment is up to date

Managed Airflow displays warnings when your environment's image approaches its end of full support date. You can use these warnings to always keep your environmentsupported.

A deprecation message is displayed on the Environment details page

Figure 1. A deprecation message is displayed on the Environment details page

Managed Airflow keeps track of the Managed Airflow image version that your environment is based on. When it approaches theend of support date, you can see a warning in the list of environments and on theEnvironment details page.

To check if your environment is up to date:

Console

  1. In Google Cloud console, go to the Environments page.
    Go to Environments
  2. In the list of environments, click the name of your environment. The Environment details page opens.
  3. Go to the Environment configuration tab.
  4. In the Image version field, one of the following messages is displayed:
    • Newest available version. Your environment image is fully supported.
    • New version available. Your environment image is fully supported and you can upgrade it to a later version.
    • Support for this image version ends in... Your environment image approaches the end of the full support period.
    • This version is not supported as of... Your environment is past the full support period.

gcloud

This functionality is not available through Google Cloud CLI. You canView suggested upgrades instead, which shows new versions that are available.

API

This functionality is not available through API. You canView suggested upgrades instead, which shows new versions that are available.

View suggested upgrades

Managed Airflow provides a list of Managed Airflow versions that you can upgrade your environment to.

To view Managed Airflow versions suggested for an upgrade:

Console

  1. In Google Cloud console, go to the Environments page.
    Go to Environments
  2. In the list of environments, click the name of your environment. The Environment details page opens.
  3. Go to the Environment configuration tab and clickUpgrade Image Version.
  4. For the list of suggested versions, click the Managed AirflowImage version drop-down menu.

gcloud

gcloud composer environments list-upgrades \
  ENVIRONMENT_NAME \
  --location LOCATION

Replace:

Example:

gcloud composer environments list-upgrades example-environment \
  --location us-central1

API

You can view available versions for a location. To do so, construct animageVersions.list API request.

For example:

// GET https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/imageVersions

Check for PyPI package conflicts

You can check if PyPI packages installed in your environment have any conflicts with preinstalled packages in the new Managed Airflow image.

A successful check means that there are no conflicts in PyPI package dependencies between the current and the specified version. However, an upgrade operation might still not be successful because of other reasons.

Console

To run an upgrade check for your environment:

  1. In Google Cloud console, go to the Environments page.
    Go to Environments
  2. In the list of environments, click the name of your environment. The Environment details page opens.
  3. Go to the Environment configuration tab, locate the Image version entry, and click Upgrade.
  4. In the Environment version upgrade dialog, in the New version drop-down list, select a Managed Airflow version that you want to upgrade to.
  5. In the PyPI packages compatibility section, clickCheck for conflicts.
  6. Wait until the check is complete. If there are PyPI package dependency conflicts, then the displayed error messages contain details about conflicting packages and package versions.

gcloud

To run an upgrade check for your environment, run theenvironments check-upgradecommand with the Managed Airflow image version. that you want to upgrade to.

gcloud composer environments check-upgrade \
  ENVIRONMENT_NAME \
  --location LOCATION \
  --image-version VERSION

Replace:

Example:

gcloud composer environments check-upgrade example-environment \
  --location us-central1 \
  --image-version composer-2.17.3-airflow-2.11.1

Example output:

Waiting for [projects/example-project/locations/us-central1/environments/
example-environment] to be checked for PyPI package conflicts when upgrading
to composer-2.17.3-airflow-2.11.1. Operation [projects/example-project/locations/
us-central1/operations/04d0e8b2-...]...done.
...

Response:
'@type': type.googleapis.com/
google.cloud.orchestration.airflow.service.v1.CheckUpgradeResponse
buildLogUri: ...
containsPypiModulesConflict: CONFLICT
pypiConflictBuildLogExtract: |-
The Cloud Build image build failed: Build failed; check build logs for
details. Full log can be found at ...
Error details: tensorboard 2.2.2 has requirement
setuptools>=41.0.0, but you have setuptools 40.3.0.

As an alternative, you can run an upgrade check asynchronously. Use the--async argument to make an asynchronous call, then check the result with the gcloud composer operations describecommand.

API

Construct an environments.checkUpgrade API request.

Specify the image version in the imageVersion field:

{
  "imageVersion": "VERSION"
}

Replace VERSION with the new version that you want to upgrade to, in the composer-a.b.c-airflow-x.y.z format.

Check for conflicts with Managed Airflow (Gen 3)

While it isn't possible to migrate your environment from Managed Airflow (Gen 2) to Managed Airflow (Gen 3) in-place, you can check for compatibility problems using an upgrade check. We recommend to do it beforemigrating to Managed Airflow (Gen 3).

To check for compatibility with Managed Airflow (Gen 3), make a regular upgrade check of your Managed Airflow (Gen 2) environment and specify a Managed Airflow (Gen 3) version as a target.

Each reported conflict can have the following types:

Console

To check for compatibility with Managed Airflow (Gen 3), use gcloud CLI or the Cloud Composer API. Google Cloud console supports only checks for PyPI package compatibility.

gcloud

To check for compatibility with Managed Airflow (Gen 3), run theenvironments check-upgradecommand and specify a Managed Airflow (Gen 3) version as a target.

gcloud composer environments check-upgrade \
  ENVIRONMENT_NAME \
  --location LOCATION \
  --image-version COMPOSER_3_VERSION

Replace:

Example:

gcloud composer environments check-upgrade example-environment \
  --location us-central1 \
  --image-version composer-3-airflow-2

Example conflict messages:

...
- message: Error validating property [core]unit_test_mode. Overriding [core]
unit_test_mode is not permitted.
type: BLOCKING
- message: Environment variables [GOOGLE_CLOUD_PROJECT] may not be overridden.
type: BLOCKING
- message: You have to specify Worker CPUs to be 0.5, 1.0 or multiples of 2.0.
type: BLOCKING
- message: The environment uses a default maintenance window, which is
different in Composer 2 and Composer 3. Unless set to a custom value, the
maintenance window will be changed to a new default after the upgrade.
type: NON_BLOCKING
...

As an alternative, you can run this check asynchronously. Use the--async argument to make an asynchronous call, then check the result with the gcloud composer operations describecommand.

API

Construct an environments.checkUpgrade API request.

Specify the image version in the imageVersion field:

{
  "imageVersion": "COMPOSER_3_VERSION"
}

Replace COMPOSER_3_VERSION with the Airflow version and buildthat you want to upgrade to, in the airflow-x.y.z-build.t format. You can use all version aliases.

Upgrade your environment

To upgrade your environment to a later version of Managed Airflow or Airflow:

Console

  1. In Google Cloud console, go to the Environments page.
    Go to Environments
  2. In the list of environments, click the name of your environment. The Environment details page opens.
  3. Go to the Environment configuration tab.
  4. Locate the Image version item and click Upgrade.
  5. From the Image version drop-down menu, select a Managed Airflow version that you want to upgrade to.
  6. Click Upgrade.

gcloud

gcloud composer environments update \
  ENVIRONMENT_NAME \
  --location LOCATION \
  --image-version VERSION

Replace:

For example:

gcloud composer environments update
  example-environment \
  --location us-central1 \
  --image-version composer-2.17.3-airflow-2.11.1

API

  1. Construct an environments.patch API request.
  2. In this request:
    1. In the updateMask parameter, specify theconfig.softwareConfig.imageVersion mask.
    2. In the request body, in the imageVersion field, specify a new version that you want to upgrade to.

For example:

// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.softwareConfig.imageVersion

  {
    "config": {
      "softwareConfig": {
        "imageVersion": "composer-2.17.3-airflow-2.11.1"
      }
    }
  }

Terraform

The image_version field in the config.software_config block controls the Managed Airflow image of your environment. In this field, specify a new Managed Airflow image.

  resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    software_config {
      image_version = "VERSION"
    }
  }
}

Replace:

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {
    software_config {
      image_version = "composer-2.17.3-airflow-2.11.1"
    }
  }
}

What's next