Log and telemetry analytics benchmark using Azure Data Explorer

The highly scalable analytics service Azure Data Explorer (ADX), which is a part of Azure Synapse Analytics, is designed for structured, semi-structured, and unstructured data. It offers customers an interactive querying experience that extracts information from the vast volume of log and telemetry data that is always expanding.

It is the ideal service for analyzing large amounts of recent and historical data in the cloud using SQL or the potent and approachable Kusto Query Language (KQL).

A crucial component of Microsoft’s digital transition is Azure Data Explorer.

All Azure services, including Azure Monitor, PlayFab, Sentinel, Microsoft 365 Defender, and many others, are used by ADX in one way or another, including troubleshooting, diagnostics, monitoring, machine learning, and data platforms.

A vast number of businesses use ADX for a wide range of scenarios, including fleet management, manufacturing, security analytics solutions, package tracking and logistics, IoT device monitoring, financial transaction monitoring, and many more.

The service has shown amazing development in recent years and is currently utilized by millions of Azure virtual machine cores.

The third generation of the Kusto engine (EngineV3) was introduced last year and is now accessible to all users who aren’t already using the most recent version as a transparent in-place update.

The storage, cache, and query execution layers all have whole new implementations in the new engine. As a result, in many mission-critical workloads, performance has increased by at least doubling.

Azure Data Explorer offers superior productivity and cost-effectiveness.

We looked for an existing telemetry and logs benchmark that meets the workload characteristics typical of what we observe with our users to better assist our users in evaluating the performance of the new engine and cost advantages of ADX:

  1. Structured, semi-structured, and unstructured data types are all present in telemetry tables.
  2. Records numbering in the billions demonstrate enormous scale.
  3. Common diagnostic and monitoring scenarios are represented by certain queries.

We partnered with and supported GigaOm to develop and run a benchmark because we could not find one that satisfied these requirements already.

This GitHub repository makes the new logs and telemetry benchmark accessible to everyone.
To learn more about ADE, feel free to contact us.

Leave a Comment