Airbyte 1.0 Event | Airbyte (original) (raw)

7,000+

daily active companies

We are pleased to announce the general availability of Airbyte 1.0, a major milestone for reliability, interoperability, ease of upgrades, and maintenance for your data pipelines. Airbyte 1.0 is immediately available to deploy, as well as for use in Airbyte Cloud.
‍

Here's what makes it prime-ready:

• Broad deployments and great support of all major use cases: with 170,000+ deployments, 7,000+ companies syncing daily, and thousands of PRs from the community, Airbyte now supports more use cases than we initially imagined. Plus, deploying it is now simpler than ever with the new single binary tool abctl that gets you started in under 5 minutes.
• Setting a reliability standard throughout the industry: we’ve introduced key reliability features in the past year that should be the foundation of any pipeline: handling large records, checkpointing, supporting very large CDC syncs, resumable full refresh, refreshes to reimport historical data with zero downtime, notifications and webhooks, monitoring sync progress, automatic detection of dropped records, and more.
• Setting a throughput performance standard: We have also significantly improved the throughput performance of our syncs: from 2MB/s to 8MB/s for API sources, and from 1MB/s in 2023 to 15MB/s for database sources. Airbyte’s throughput performances are now higher than the competition, but our ambitions don’t stop there: Airbyte should never be the bottleneck.
• Fitting your workflows and data infrastructure: Airbyte integrates into your existing infrastructure, no matter what you’re running, i.e. all major orchestration, transformation, metadata, chunking and embedding tools. Airbyte also supports all major destinations (warehouses, lakes, databases and vector databases), including Databricks (newly certified). You can also manage pipelines whatever your production workflows are, through the UI, API, Terraform Provider and even Python library PyAirbyte. Airbyte’s version upgrades are now seamless, and shouldn’t impact your workflow either.

Here's a quick demo running through the abctl deployment, Helm charts and Terraform Provider:

Airbyte 1.0 delivers the level of quality and performance an industry standard would have. Whether you're a data engineer or a Fortune 500, Airbyte can scale your data pipelines. Rocketships like Perplexity.ai, Monday.com, Calendly, now use Airbyte 1.0 to power their production data pipelines.

Bonus: We’ve now also released our first official Airbyte course.

We are excited to share that Airbyte Self-Managed Enterprise, the engine for self-serve data platforms, is now generally available. Self-Managed Enterprise is specifically tailored to scale comfortably to any workload in air gapped, isolated environments. Self-Managed Enterprise extends on Airbyte 1.0 by introducing new classes of functionality, while ensuring data never leaves your infrastructure:
‍

1. Multitenancy & Role-Based Access: Manage multiple teams and projects within a single Airbyte deployment. This feature empowers citizen developers to discover and consume data across your organization, all while managing team access from a single pane of glass.
2. PII Masking: Protect sensitive information by hashing personally identifiable information (PII) as it moves through your pipelines. This ensures compliance with privacy regulations and allows for greater pipeline consolidation within Airbyte.
3. Enterprise Support with SLAs: Airbyte Self-Managed Enterprise comes with dedicated support and guaranteed service level agreements (SLAs), ensuring that your data movement infrastructure remains reliable and performant, and expert assistance is available when needed.

Whether you need a fully isolated deployment of Airbyte, or prefer a fully managed solution, Airbyte has you covered. You may also purchase Airbyte Enterprise using your committed cloud provider spend via the AWS or GCP marketplaces. To learn more about our enterprise solutions, reach out to our team.

With 1.0, we’re launching our Connector Marketplace with hundreds of connectors already contributed by the community. While the Airbyte team doesn’t maintain these connectors like our Certified connectors, we provide success rate and usage indicators to help you gauge their reliability and maturity. All the marketplace connectors, built with our low-code Connector Builder, can be used as-is or customized to your needs. And we’re just getting started! Publishing a new connector or the edits to existing one is now available at a click of the button within the Connector Builder.

‍

In addition to the Marketplace, Airbyte 1.0 features our brand new AI Assist that will build the connector for you! You just need to input the API docs link and specify the streams you would like to have, and the AI generates a working connector that you can tweak and deploy directly. It’s now easier than ever to expand your data ecosystem.

AI Assist (built in collaboration with Fractional AI) is available in open beta, only on Airbyte Cloud for now, as it relies on our own resources. It will soon become available in self-hosted Airbyte Open Source and Airbyte Enterprise.

Airbyte’s Hacktoberfest 2024 will be focused on expanding our marketplace connectors to help it reach 1,000. Join the competition and get a chance to win $20,000+ worth of prizes!

In today’s data-driven world, accurate data is critical for survival, especially as AI takes center stage. Companies face an unsustainable challenge with growing data sources and maintaining in-house connectors, consuming 44% of data teams' time and resources. Given closed-source solutions are proving inflexible and require complementary in-house solutions, data teams need to adopt an open data movement infrastructure now, before too much technical debt is accumulated and they miss the AI wave. That’s where Airbyte comes in with our open-source approach. Our vision is to become the data movement standard for the new AI era all companies will have to adopt if they want a chance to survive and win.

Airbyte already supports major vector databases, unstructured data sources, and AI-driven transformations like document conversion, embedding, and chunking. PyAirbyte, our Python library, has been leveraged by our AI community extensively and is released today in General Availability too. Airbyte has already built the foundation for Enterprise AI success. Here’s a quick demo:

This is only the beginning for us, as we have much more in store. Embrace the AI future with Airbyte and stay ahead in the AI revolution!

The Airbyte community is deeply connected to a network of data-focused companies and communities pushing the boundaries of data and AI engineering. We’ve established top-tier partnerships with industry leaders like Snowflake, AWS, GCP, dbt, Astronomer, Dagster, and Prefect, as well as with AI innovators like LangChain, LlamaIndex, OpenAI, Cohere, and all significant data destinations.

These partnerships are not just about collaboration but are central to advancing the future of data and AI. You can explore the complete list of our partners on our Partners page. If you want to join Airbyte forces, visit our Partner Portal for more information.

For insights into the future of data and AI, check out our panel discussion below featuring the CEOs of Airbyte, dbt, Dagster, and LangChain. They discuss the next AI era of data-driven innovation. The Airbyte community is at the forefront of this revolution, and we invite you to join us as we continue to innovate and grow together.

Airbyte 1.0 marks a significant step forward in addressing the data movement challenges your organization faces for their data and AI needs. While we’re still early in this journey, today is a turning point - Airbyte is now enterprise-grade and primetime-ready, poised to become the standard for the emerging AI era.

So, what’s next? The easiest way to show how vision is to illustrate it!

As you can see, much to do!