Microsoft and F5 join forces on OpenTelemetry with Apache Arrow in Rust - Microsoft Open Source Blog (original) (raw)
Microsoft and F5 are collaborating on Phase 2 of the OpenTelemetry with Apache Arrow project. This project is developing a high-performance telemetry pipeline integrated with the Apache Arrow ecosystem, with a range of applications for small- and large-scale uses. We are creating new telemetry pipeline libraries in the Rust language, and we aim to deliver this functionality through integration with the OpenTelemetry Collector.
In Phase 1 of the project, we developed OpenTelemetry-Arrow Protocol (OTAP) libraries meant for optimizing network usage between Collectors based on Apache Arrow IPC streams. In this phase of the project, we are developing OTAP libraries for optimizing the cost of telemetry inside Collectors, in terms of CPU usage, using Apache Arrow data frames as the unit of transport between producers and consumers.
We are bringing OpenTelemetry use-cases into the Apache Arrow ecosystem, with a promise of full compatibility. OpenTelemetry with Apache Arrow gives OpenTelemetry users a way to exchange OpenTelemetry data with a wide range of data-oriented applications. We think it will be good for OpenTelemetry to have access to the broader Apache Arrow and Rust ecosystems.
A look back: Phase 1 success
Phase 1 of OpenTelemetry with Apache Arrow project began with a reference implementation of OTAP streams for the OpenTelemetry Collector. Now included in the OpenTelemetry Collector-Contrib distribution, these components act as drop-in replacements for the OTLP receiver and exporter. Anywhere OTLP requests are being used for bulk telemetry transport between Collectors, a “compression bridge” using OTAP streams instead could be established to achieve substantially better compression.
For users, this means sending 30 to 70% less data over the network, simply from using OTAP instead of OTLP. The network savings result from translating OTLP data out of its hierarchical representation, based on Protocol Buffers, into a tabular representation based on Apache Arrow, the results achieved first through sorting and de-duplication, then through Arrow IPC encoding optimizations.
Phase 1 was just the beginning. It proved the power of combining OpenTelemetry and Apache Arrow and set the stage for Phase 2.
Phase 2: Enter Rust and Arrow
In Phase 2, Microsoft and F5 are building a fully native, embeddable OpenTelemetry pipeline powered by Rust and Apache Arrow. The goal? Dramatic improvements in CPU utilization, memory efficiency, throughput, and latency—all of which contribute to a more efficient data processing ecosystem.
The decision to go “all in” with Rust has stirred excitement—and a bit of controversy—among the technical community. After all, OpenTelemetry’s Collector project is traditionally built using Golang. So why Rust?
Rust and Apache Arrow are like best friends in the data processing world. Both technologies complement each other profoundly, making Rust the ideal choice for Phase 2 development. Here are just a few reasons:
- Memory safety and performance:
Rust’s commitment to memory safety ensures that data pipelines are both robust and secure, minimizing risks like data corruption or segmentation faults. Combine this with Rust’s performance characteristics—low overhead, zero-cost abstractions—and you have a perfect match for efficiently handling high-speed, high-volume telemetry data. - Columnar data processing:
Apache Arrow is the de facto framework for zero-copy, column-oriented data processing. It is widely adopted across modern data lakes, supporting systems like Parquet, Delta Lake, and Iceberg. Native integration with Arrow means you can exchange data seamlessly and efficiently with powerful analytics engines and databases. - Zero-copy design:
Apache Arrow’s zero-copy paradigm aligns with Rust’s emphasis on direct memory access. Together, these technologies eliminate serialization and deserialization overhead, making it possible to handle large data streams efficiently without wasting CPU cycles. - Thriving ecosystem:
The Rust ecosystem around Apache Arrow is thriving. By choosing Rust for Phase 2, F5 and Microsoft are joining this vibrant community, unlocking integrations with powerful libraries and frameworks such as Apache DataFusion.
OpenTelemetry Collector integration
Despite the buzz around Rust, Phase 2 doesn’t mean a departure from the OpenTelemetry Collector. In fact, F5 and Microsoft are taking careful steps to ensure compatibility between Go and Rust pipelines, by safeguarding embeddability, prioritizing strict memory controls, and using a thread-per-core runtime model.
We will continue supporting our existing OpenTelemetry Collector components and developing our Golang OTAP reference implementation, and we will ensure the OpenTelemetry Collector can run and manage OTAP pipelines.
What’s next?
While we expect dramatic improvements, we can’t be sure without benchmarks. Our plans call for extensive performance testing, especially cost and reliability benchmarks comparing Go and Rust pipelines over basic functions. Until the results are in, our project will be considered experimental. The community has shown a strong level of interest, and one thing is clear—the OpenTelemetry with Apache Arrow project is helping shape the future of distributed observability systems.
Get involved
The OpenTelemetry with Apache Arrow project is looking for interested developers, find us at our project repository or in our weekly meetings. Learn more about Rust projects at Microsoft, and find more about Microsoft’s contributions to OpenTelemetry on our blog.
Here’s a deep-dive by Microsoft engineer Raki Rahman showing how to work with OTel-Arrow data using a T-SQL notebook on Microsoft Fabric.
Learn more about OpenTelemetry with Apache Arrow Phase 2 on the OpenTelemetry blog.
Enabling Microsoft products and services to bring choice, technology, and community to our customers
Joshua MacDonald is an OpenTelemetry contributor working in the observability industry. On the side, he operates a community water system."