Reference for package extras — Airflow Documentation (original) (raw)
These providers extras are simply convenience extras to install provider packages so that you can install the providers with simple command - including provider package and necessary dependencies in single command, which allows PIP to resolve any conflicting dependencies. This is extremely useful for first time installation where you want to repeatably install version of dependencies which are ‘valid’ for both airflow and providers installed.
with a consistent set of dependencies based on constraint files provided by Airflow Community at the time 2.5.1 version was released.
Note, that this will install providers in the versions that were released at the time of Airflow 2.5.1 release. You can later upgrade those providers manually if you want to use latest versions of the providers.
Apache Software extras¶
These are extras that add dependencies needed for integration with other Apache projects (note that apache.atlas andapache.webhdfs do not have their own providers - they only install additional libraries that can be used in custom bash/python providers).
| extra | install command | enables |
|---|---|---|
| apache.atlas | pip install 'apache-airflow[apache.atlas]' | Apache Atlas |
| apache.beam | pip install 'apache-airflow[apache.beam]' | Apache Beam operators & hooks |
| apache.cassandra | pip install 'apache-airflow[apache.cassandra]' | Cassandra related operators & hooks |
| apache.drill | pip install 'apache-airflow[apache.drill]' | Drill related operators & hooks |
| apache.druid | pip install 'apache-airflow[apache.druid]' | Druid related operators & hooks |
| apache.hdfs | pip install 'apache-airflow[apache.hdfs]' | HDFS hooks and operators |
| apache.hive | pip install 'apache-airflow[apache.hive]' | All Hive related operators |
| apache.kylin | pip install 'apache-airflow[apache.kylin]' | All Kylin related operators & hooks |
| apache.livy | pip install 'apache-airflow[apache.livy]' | All Livy related operators, hooks & sensors |
| apache.pig | pip install 'apache-airflow[apache.pig]' | All Pig related operators & hooks |
| apache.pinot | pip install 'apache-airflow[apache.pinot]' | All Pinot related hooks |
| apache.spark | pip install 'apache-airflow[apache.spark]' | All Spark related operators & hooks |
| apache.sqoop | pip install 'apache-airflow[apache.sqoop]' | All Sqoop related operators & hooks |
| apache.webhdfs | pip install 'apache-airflow[apache.webhdfs]' | HDFS hooks and operators |
External Services extras¶
These are extras that add dependencies needed for integration with external services - either cloud based or on-premises.
| extra | install command | enables |
|---|---|---|
| airbyte | pip install 'apache-airflow[airbyte]' | Airbyte hooks and operators |
| alibaba | pip install 'apache-airflow[alibaba]' | Alibaba Cloud |
| amazon | pip install 'apache-airflow[amazon]' | Amazon Web Services |
| asana | pip install 'apache-airflow[asana]' | Asana hooks and operators |
| atlassian.jira | pip install 'apache-airflow[atlassian.jira]' | Jira hooks and operators |
| azure | pip install 'apache-airflow[microsoft.azure]' | Microsoft Azure |
| cloudant | pip install 'apache-airflow[cloudant]' | Cloudant hook |
| databricks | pip install 'apache-airflow[databricks]' | Databricks hooks and operators |
| datadog | pip install 'apache-airflow[datadog]' | Datadog hooks and sensors |
| dbt.cloud | pip install 'apache-airflow[dbt.cloud]' | dbt Cloud hooks and operators |
| dingding | pip install 'apache-airflow[dingding]' | Dingding hooks and sensors |
| discord | pip install 'apache-airflow[discord]' | Discord hooks and sensors |
| pip install 'apache-airflow[facebook]' | Facebook Social | |
| pip install 'apache-airflow[google]' | Google Cloud | |
| hashicorp | pip install 'apache-airflow[hashicorp]' | Hashicorp Services (Vault) |
| opsgenie | pip install 'apache-airflow[opsgenie]' | OpsGenie hooks and operators |
| pagerduty | pip install 'apache-airflow[pagerduty]' | Pagerduty hook |
| plexus | pip install 'apache-airflow[plexus]' | Plexus service of CoreScientific.com AI platform |
| qubole | pip install 'apache-airflow[qubole]' | Enable QDS (Qubole Data Service) support |
| salesforce | pip install 'apache-airflow[salesforce]' | Salesforce hook |
| sendgrid | pip install 'apache-airflow[sendgrid]' | Send email using sendgrid |
| segment | pip install 'apache-airflow[segment]' | Segment hooks and sensors |
| slack | pip install 'apache-airflow[slack]' | Slack hooks and operators |
| snowflake | pip install 'apache-airflow[snowflake]' | Snowflake hooks and operators |
| tableau | pip install 'apache-airflow[tableau]' | Tableau hooks and operators |
| tabular | pip install 'apache-airflow[tabular]' | Tabular hooks |
| telegram | pip install 'apache-airflow[telegram]' | Telegram hooks and operators |
| vertica | pip install 'apache-airflow[vertica]' | Vertica hook support as an Airflow backend |
| yandex | pip install 'apache-airflow[yandex]' | Yandex.cloud hooks and operators |
| zendesk | pip install 'apache-airflow[zendesk]' | Zendesk hooks |
Locally installed software extras¶
These are extras that add dependencies needed for integration with other software packages installed usually as part of the deployment of Airflow.
| extra | install command | enables |
|---|---|---|
| arangodb | pip install 'apache-airflow[arangodb]' | ArangoDB operators, sensors and hook |
| docker | pip install 'apache-airflow[docker]' | Docker hooks and operators |
| elasticsearch | pip install 'apache-airflow[elasticsearch]' | Elasticsearch hooks and Log Handler |
| exasol | pip install 'apache-airflow[exasol]' | Exasol hooks and operators |
| github | pip install 'apache-airflow[github]' | GitHub operators and hook |
| influxdb | pip install 'apache-airflow[influxdb]' | Influxdb operators and hook |
| jenkins | pip install 'apache-airflow[jenkins]' | Jenkins hooks and operators |
| mongo | pip install 'apache-airflow[mongo]' | Mongo hooks and operators |
| microsoft.mssql | pip install 'apache-airflow[microsoft.mssql]' | Microsoft SQL Server operators and hook. |
| mysql | pip install 'apache-airflow[mysql]' | MySQL operators and hook |
| neo4j | pip install 'apache-airflow[neo4j]' | Neo4j operators and hook |
| odbc | pip install 'apache-airflow[odbc]' | ODBC data sources including MS SQL Server |
| openfaas | pip install 'apache-airflow[openfaas]' | OpenFaaS hooks |
| oracle | pip install 'apache-airflow[oracle]' | Oracle hooks and operators |
| postgres | pip install 'apache-airflow[postgres]' | PostgreSQL operators and hook |
| presto | pip install 'apache-airflow[presto]' | All Presto related operators & hooks |
| redis | pip install 'apache-airflow[redis]' | Redis hooks and sensors |
| samba | pip install 'apache-airflow[samba]' | Samba hooks and operators |
| singularity | pip install 'apache-airflow[singularity]' | Singularity container operator |
| trino | pip install 'apache-airflow[trino]' | All Trino related operators & hooks |
Other extras¶
These are extras that provide support for integration with external systems via some - usually - standard protocols.
| extra | install command | enables | Preinstalled |
|---|---|---|---|
| common.sql | pip install 'apache-airflow[common.sql]' | Core SQL Operators | |
| ftp | pip install 'apache-airflow[ftp]' | FTP hooks and operators | |
| grpc | pip install 'apache-airflow[grpc]' | Grpc hooks and operators | |
| http | pip install 'apache-airflow[http]' | HTTP hooks, operators and sensors | |
| imap | pip install 'apache-airflow[imap]' | IMAP hooks and sensors | |
| jdbc | pip install 'apache-airflow[jdbc]' | JDBC hooks and operators | |
| microsoft.psrp | pip install 'apache-airflow[microsoft.psrp]' | PSRP hooks and operators | |
| microsoft.winrm | pip install 'apache-airflow[microsoft.winrm]' | WinRM hooks and operators | |
| papermill | pip install 'apache-airflow[papermill]' | Papermill hooks and operators | |
| sftp | pip install 'apache-airflow[sftp]' | SFTP hooks, operators and sensors | |
| sqlite | pip install 'apache-airflow[sqlite]' | SQLite hooks and operators | |
| ssh | pip install 'apache-airflow[ssh]' | SSH hooks and operators |