Managed Service for Apache Airflow versioning overview (original) (raw)

This page describes Managed Service for Apache Airflow versioning and lists differences between major Managed Airflow versions.

Major versions of Managed Airflow

Managed Airflow has the following major versions:

• Managed Airflow (Gen 3). This version has simplified network setup and hides infrastructure components including the environment's cluster and environment's dependencies on other services.
• Managed Airflow (Gen 2). In this version, the environment's cluster scales automatically to meet the resource demands.
• Managed Airflow (Legacy Gen 1). Has manual environment scaling, the infrastructure is deployed to your projects and networks.

Comparison of Managed Airflow versions

The following table lists major differences between Managed Airflow (Legacy Gen 1), Managed Airflow (Gen 2), and Managed Airflow (Gen 3).

Comparison of Airflow versions

Managed Airflow (Gen 3) provides environments with Airflow 2 and Airflow 3.

Support for Managed Airflow (Gen 3) features

The following table summarizes Managed Airflow (Gen 3) features thataren't yet supported in Airflow 3.

Known issues in Airflow 3

The following known issues have been identified in Airflow 3:

• Warning messages in the logs of Airflow workloads. These messages can be safely ignored.
• The airflowctl utility can't be easily used from headless environments such as Airflow workers in Managed Airflow or Docker containers. This is a limitation of the utility. For more information about this limitation, seeIssue 58230in the Airflow repository on GitHub.

Support for Airflow 3 features

The following table lists Airflow 3 features and their support:

To run Apache Airflow, Managed Airflow builds Docker images that bundle Airflow releases with other common binaries and Python libraries.

Managed Airflow images include Airflow modifications that are specific to Managed Airflow and unsuitable for the upstream Airflow codebase. Each Managed Airflow image contains preinstalled packages along with Managed Airflow-specific Airflow changes.

Google periodically releasesnew Managed Airflow images:

• In Managed Airflow (Gen 3), images include new builds of supported Airflow versions. You can select an Airflow version and a build that you want to use. Your environment gets Managed Airflow improvements automatically.
• In Managed Airflow (Gen 2) and Managed Airflow (Legacy Gen 1), images include Managed Airflow improvements and new builds of supported Airflow versions. To get Managed Airflow improvements, you upgrade your environment to a later version of Managed Airflow.

Version deprecation and support

Composer 3

Google supports an Airflow build for a period of time after it was released:

• 0-12 months from the release date: Managed Airflow (Gen 3) environments running this Airflow build are supported.
• 12+ months from the release date: Managed Airflow (Gen 3) environments running these versions are unsupported.
  After an Airflow build is no longer supported, environments running this Airflow build are also unsupported and entirely user-managed.

Environments work with supported and unsupported Airflow builds in the following way:

• You can create environments with all Airflow builds that are supported.
• You can still use an environment with an Airflow build that is no longer supported. The environment still works, you can keep using it, and you canupgrade it to asupported Airflow build.
• In some cases, an upgrade to a later Airflow version or build is the only available solution for a security vulnerability.

Composer 2

Google supports Managed Airflow (Gen 2) versions for a period of time after the version's release:

• 0-12 months from the release date: Managed Airflow (Gen 2) environments running these versions are supported.
• 12+ months from the release date: Managed Airflow (Gen 2) environments running these versions are unsupported.

You can use Managed Airflow versions after the date they are no longer supported. For example, if your environment is based on a Managed Airflow version that is past the support period, the environment still works and you can keep using it. In this case, we recommend you to upgrade your environment to asupported version.

Composer 1

New Managed Airflow (Legacy Gen 1) versions were released until the end of March 2023. All released Managed Airflow (Legacy Gen 1) versions follow the same version deprecation policy as Managed Airflow (Gen 2) and are supported according to it.

Version schemas

This section describes Airflow and Managed Airflow version schemas and version aliases.

Airflow version schema

Airflow follows the semantical software versioning schema. Each version of Airflow has a major, minor, and patch version.

Composer 3

In Managed Airflow (Gen 3), new Airflow versions and builds become available as the time goes.

Your environment can function with all previous Airflow versions and builds, and receives updates to its infrastructure components automatically. Each version and build is supported as described inVersion deprecation and support, and you can choose to upgrade your Airflow version to a later version or build.

Composer 2

For a list of Airflow versions supported in each Managed Airflow version, see the list of Managed Airflow versions.

• In each release, Managed Airflow (Gen 2) supportstwo minor versions of Airflow 2.
  For example, Managed Airflow 2.4.0 supports Airflow 2.4.* and Airflow 2.5.*.
• For each minor version of Airflow 2, Managed Airflowsupports one patch version.
  For example, for Airflow 2.4.*, Managed Airflow supports Airflow 2.4.3.
• When a new patch version of Airflow 2 becomes available in Managed Airflow, the corresponding release has two patch versions of Airflow 2 available for the same minor version of Airflow. The previous patch version is then removed in one of the next releases.
  For example, Managed Airflow 2.3.4 supports both Airflow 2.5.1 and Airflow 2.5.3.
• Managed Airflow can skip some Airflow patch versions between releases of Managed Airflow. In addition, if there are functional, quality, or performance issues with a specific patch version of Airflow, Managed Airflow can also skip this version.
  For example, Airflow 2.1.3 is skipped and Airflow 2.1.4 is available.
• A Managed Airflow release running a stable Airflow version can include Airflow updates that are backported from a later Airflow version.

Composer 1

The latest version of Airflow supported in Managed Airflow (Legacy Gen 1) is Airflow 2.4.3.

Only Managed Airflow (Legacy Gen 1) supports Airflow 1.10.15, which is the latest released version of Airflow 1.

Managed Airflow version schema

Composer 3

Managed Airflow (Gen 3) versioning schema captures Airflow versions and builds:

composer-3-airflow-x.y.z-build.t

Where:

• composer-3 is the major version of Managed Airflow.
• airflow-x.y.z.build.t is the Airflow version and build. Every next release of the same Airflow version in Managed Airflow (Gen 3) has a higher build number. The build number is defined by Managed Airflow and not by the Airflow open source project.

Composer 2

Managed Airflow (Gen 2) image versioning schema captures Managed Airflow and Airflow versions:

composer-2.b.c-airflow-x.y.z

Where:

• composer-2.b.c denotes the minor and patch version of Managed Airflow-managed customizations in Managed Airflow (Gen 2). As a general rule, minor versions of Managed Airflow correspond to significant changes in functionality compared to the previous minor version, such as when the available set of Airflow versions is changed. Patch versions are updated in subsequent releases.
• airflow-x.y.z is the Airflow release, it denotes the major, minor, and patch version of Airflow that's used in a specific Managed Airflow image.

Composer 1

Managed Airflow (Legacy Gen 1) image versioning schema captures Managed Airflow and Airflow versions:

composer-1.b.c-airflow-x.y.z

Where:

• composer-1.b.c is the version of Managed Airflow-managed customizations
• airflow-x.y.z is the Airflow release

Version aliases

Version aliases work in the following way:

• Managed Airflow resolves the version alias to the full format when it creates your environment:
  • All version aliases become composer-3-airflow-x.y.z-build.t in Managed Airflow (Gen 3).
  • All version aliases become composer-a.b.c-airflow-x.y.z in Managed Airflow (Gen 2) and Managed Airflow (Legacy Gen 1).
• In Managed Airflow (Gen 3), using a version alias, such ascomposer-3-airflow-x.y does not automatically upgrade the Airflow version and build of your environment. The environment stays on the same Airflow build and version until you upgrade it. The environment still receives automatic infrastructure upgrades, as usual.
• In Managed Airflow (Gen 2) and Managed Airflow (Legacy Gen 1), using a version alias, such ascomposer-a-airflow-x.y.z does not provide automatic upgrades for an environment. The environment stays on the same Managed Airflow and Airflow version until you upgrade it.

Managed Airflow (Gen 3) version aliases

Managed Airflow (Gen 3) supports the following version aliases:

Managed Airflow (Gen 2) version aliases

Managed Airflow (Gen 2) supports the following version aliases:

Managed Airflow (Legacy Gen 1) version aliases

Managed Airflow (Legacy Gen 1) supports the following version aliases:

The following version aliases point to Managed Airflow (Gen 3). You can still use them, but consider switching to aliases for your version of Managed Airflow.

• composer-latest-airflow-x.y is equal to composer-3-airflow-x.y.
• composer-latest-airflow-x.y.z is equal to composer-3-airflow-x.y.z.

Version arguments in Google Cloud CLI

When you create or upgradea Managed Airflow environment using Google Cloud CLI, you can specify Managed Airflow and Airflow versions:

Composer 3

• --image-version argument specifies the Airflow version and build in thecomposer-3-airflow-x.y.z-build.t format. You can useversion aliases, which are resolved by Google Cloud CLI into the full format.
• --airflow-version specifies the Airflow version in the following formats:
  • The x.y.z format is for the latest build of the specified version of Airflow. This format is equal to the composer-3-airflow-x.y.zversion alias.
  • The x.y format is for the latest build of Airflow available withinx.y major and minor versions. This format is equal to thecomposer-3-airflow-x.y version alias.
  • x.y.z-build.t is the specified version and build of Airflow. This format is equal to composer-3-airflow-x.y.z-build.t.

Composer 2

• --image-version argument specifies Managed Airflow (Gen 2) and Airflow versions:
  • composer-a.b.c-airflow-x.y.z format creates an environment with the specified versions of Managed Airflow and Airflow.
  • Version aliases first resolve to the composer-a.b.c-airflow-x.y.z format and then create an environment with the specified versions of Managed Airflow (Gen 2) and Airflow.
• --airflow-version specifies the Airflow version in the following formats:
  • The x.y.z format is for the latest version of Managed Airflow (Gen 2) and the specified version of Airflow. This format is equal to the composer-2-airflow-x.y.z version alias.
  • The x.y format is for the latest version of Managed Airflow (Gen 2) and the latest version of Airflow available within x.y major and minor versions. This format is equal to the composer-2-airflow-x.yversion alias.

Composer 1

• --image-version argument specifies Managed Airflow (Legacy Gen 1) and Airflow versions:
  • composer-a.b.c-airflow-x.y.z format creates an environment with the specified versions of Managed Airflow and Airflow.
  • Version aliases first resolve to the composer-a.b.c-airflow-x.y.z format and then create an environment with the specified versions of Managed Airflow (Legacy Gen 1) and Airflow.
• --airflow-version specifies the Airflow version in the following formats:
  • The x.y.z format is for the latest version of Managed Airflow (Legacy Gen 1) and the specified version of Airflow. This format is equal to the composer-1-airflow-x.y.z version alias.
  • The x.y format is for the latest version of Managed Airflow (Legacy Gen 1) and the latest version of Airflow available withinx.y major and minor versions. This format is equal to thecomposer-1-airflow-x.y version alias.

What's next

• Managed Airflow images
• Create an environment
• Upgrade an existing Managed Airflow environment