Understanding telemetry pipelines | NXLog Blog (original) (raw)
Back in the day, Gordon Moore made relatively accurate observations and projections about the exponential growth of transistors on semiconductors. It still amazes me, yet very few predicted the incredible growth of system interconnectedness and the vast amount of data it generates. It is estimated that 90% of all data was created in the last last two years. Given that everything is connected, the need for telemetry is growing at an unprecedented rate, and thus, the need to efficiently channel and manage telemetry data has also grown.
Telemetry is the ecosystem built around collecting metrics remotely. Telemetry collects telemetry data; the system or chain of systems that collects this data is called a telemetry pipeline or a telemetry data pipeline. In this post, we look into that.
What is telemetry and telemetry pipeline?
In the English language, telemetry is a foreign word, originating from the French télémétre, which is a compound word consisting of télé (meaning far) and métre (meaning a device for measuring). The essence of the word implies remote observation and collection of metrics. Implementing the definition of the expression in the modern IT world, telemetry means a set of tools or a framework that collects and transmits telemetry data from sensors on devices from remote or inaccessible places. In the metrics collection system, the telemetry pipeline is responsible for collecting and shifting data to its destination, and you could consider it as a subset of the observability pipeline, which also includes tools for data visualization, analysis, alerting, and so on.
Telemetry pipelines can handle diverse data types (metrics, logs, traces) and provide real-time or near-real-time data processing. A telemetry pipeline works based on the same principles as a "conventional" log data pipeline. The main differences are the type of data it shifts and the time sensitivity of the data. Given these differences, they need different handling, priorities, and routes.
What is the importance of collecting metrics?
Imagine when you operate a complex industrial control system, for example, in a crude oil refinery or a windmill farm on the open sea, to mention a few exciting cases when telemetry and the data gathered from it could be paramount for your operation. An environment where the smallest resonance of the system counts when even the pressure in the smallest pipe is critical for the safety of those working there. On the other hand, you might also be operating a long production line where downtime costs a lot, and you want to minimize the risk of losing money.
According to a Gartner report, telemetry pipelines are projected to be used in 40% of logging solutions by 2026 due to the complexity of distributed architectures. Read more on Gartner.
A well-designed and implemented telemetry strategy and a thoughtfully crafted telemetry pipeline are the two fundamental building pillars of your metrics collection. They also play a crucial role in your IT security and operational continuity. A telemetry data pipeline enables real-time monitoring and analysis of system performance to identify and resolve issues swiftly.
- It helps optimize system resources and improve overall reliability and efficiency.
- It aids mission-critical elements of your operational chain.
- It facilitates proactive maintenance and observability by providing timely insights with relevant, fresh data.
What is the secret to building a telemetry pipeline?
Having a dedicated toolchain for your your telemetry operation, accommodating a safe and swift route to your telemetry data has many advantages. When planning and building a telemetry pipeline, there are a few principles to keep in mind.
Toolchain selection for collection
Choose tools that accommodate efficient data collection and support filtering mechanisms to ensure only relevant telemetry data is collected.
Data filtering and trimming
Early in the telemetry pipeline, filter and trim the telemetry data to eliminate redundant information entering the rest of the "journey". This step reduces log noise and improves the efficiency of data flow. In addition, many SIEM and analytics system charge their customers by the amount of data ingested, so this step can also significantly lower costs.
Data normalization
Standardize the collected telemetry data by normalizing it into a common format that makes it easier to compare and analyze it. Normalizing your telemetry data also helps in later stages by easing correlation.
Relaying and packet prioritization
Relay the telemetry data with efficient routing and packet prioritization to ensure that real-time and critical data is shifted first, keeping latency minimal when dealing with time sensitive data.
Data formatting for destinations
Format your data to ensure that the output format is compatible with the target analytics or visualization platform. This part is key to seamless data processing.
When is collecting telemetry data beneficial?
Telemetry data collection is particularly valuable in environments with complex, distributed, or mission-critical systems. Here are some scenarios where telemetry pipelines shine:
Monitoring distributed systems
- Without telemetry pipelines — Monitoring each part of a distributed system separately leads to gaps in visibility. A microservice failure may not be linked to a database issue, causing delayed fixes.
- Risks — Increased downtime, missed issues, and inefficient system management.
- With telemetry pipelines — Centralized monitoring across services allows fast detection and resolution of issues, improving system uptime and performance.
For more insights into the challenges of handling telemetry data, check out this DevOps article.
Managing critical infrastructure
- Without telemetry pipelines — Independent monitoring systems in healthcare, finance or IT security can miss critical failures, risking safety and compliance.
- Risks — Delayed detection of failures, potential safety risks, and regulatory penalties.
- With telemetry pipelines — Centralized data collection ensures real-time issue detection and compliance, reducing risks and improving reliability.
Optimizing high-volume data processing
- Without telemetry pipelines — High data volumes overwhelm traditional systems, making it hard to filter relevant insights and increasing costs.
- Risks — Wasted resources, missed insights, and delayed responses.
- With telemetry pipelines — Telemetry pipelines filter and route relevant data efficiently, reducing costs and enabling faster decision-making.
In summary, telemetry pipelines are essential for ensuring the smooth operation of complex systems, enhancing performance, and enabling proactive management.
Telemetry pipelines versus observability pipelines
An interesting section title, I must say. Many people seem to be confused and there are some who think these concepts are the same and are interchangeable. Well, that is incorrect. Let’s see why.
Telemetry pipelines focus on collecting and processing data like logs, metrics, and traces from different parts of a system. Observability pipelines do more by combining this data with other information to give a full picture of how the system is doing. Telemetry pipelines are a key part of observability pipelines, providing the data needed to understand and monitor the system effectively. Simply put, telemetry pipelines gather the data, and observability pipelines works with that data to help you visualize and understand what’s happening in your system.
Final thoughts on telemetry pipelines
Telemetry pipelines are an important part of modern observability systems because they help the safe and swift transition of your telemetry data from your remote sources. They are responsible for the collection and transmission of the information needed to keep everything running smoothly and help identify and fix issues early. While they’re just one piece of the puzzle, they’re crucial for making sure your systems are in their best shape. As IT systems become more connected and complex, having a solid telemetry strategy to collect remote metrics will only become more important.