Feature Request for Azure Monitor Metrics Data Plane Batch API Support

[ashishbans]

Component(s)

receiver/azuremonitor

Is your feature request related to a problem? Please describe.

Currently, the OpenTelemetry Collector's Azure integration uses the standard Azure Resource Manager (ARM) APIs for metrics collection, which has significant limitations:

• Low query limits (12,000 API calls per hour)
• Requirement to make separate API calls for each resource
• Higher risk of throttling when monitoring many Azure resources
• Lower performance for large-scale monitoring scenarios
• These limitations create challenges for users monitoring large Azure environments with multiple resources, as they frequently encounter throttling issues and inefficient data collection.

Describe the solution you'd like

Implement support for the new Azure Monitor Metrics Data Plane API in the OpenTelemetry Collector. This API offers several significant advantages:

• Higher query limits: Up to 360,000 API calls per hour (30x improvement over ARM API)
• Batch capabilities: Collect metrics for up to 50 resource IDs in a single API call
• Improved efficiency: Reduced number of API calls for the same amount of metric data
• Regional endpoints: Using region-specific endpoints for better performance
• The implementation should allow users to:
• Configure the collector to use the new Metrics Data Plane API instead of ARM APIs
• Specify Azure resource IDs to be monitored in batches
• Take advantage of the higher throughput and reduced throttling risk

Describe alternatives you've considered

No response

Additional context

The Azure Monitor Metrics Data Plane API is Generally Available and represents a significant improvement in Azure monitoring capabilities. Key technical details:

• API endpoint format: {endpoint}/subscriptions/{subscriptionId}/metrics:getBatch?metricnamespace={metricnamespace}&metricnames={metricnames}&api-version=2023-10-01
• Uses regional endpoints (e.g., https://eastus.metrics.monitor.azure.com)
• Authenticated via Azure Active Directory with the scope https://metrics.monitor.azure.com/.default
• Supports comprehensive filtering and aggregation capabilities similar to existing ARM APIs
• This feature would significantly improve the experience for users monitoring large Azure environments and ensure the OpenTelemetry Collector remains a competitive option for Azure monitoring.
• How to migrate from the metrics API to the getBatch API
• Azure Monitor: Announcing GA of Azure Metrics Data Plane API
• Azure Monitor Metrics Batch API Documentation

[github-actions]

Pinging code owners:

• receiver/azuremonitor: @nslaughter @celian-garcia

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[celian-garcia]

Thank you for this very detailed feature request. I'm actually planning to propose a PR from my fork that uses the getBatch API. This is the fork that we are using in my company for months already with good success!
I didn't know about this new API though. This is interesting, do you have go libraries associated?

[ashishbans]

Hey @celian-garcia , glad to know that your company already using getBatch API

It is same. Here is the go libraries for Batch API
https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/monitor/azquery

[celian-garcia]

Ok we're actually using github.com/Azure/azure-sdk-for-go/sdk/monitor/query/azmetrics
For the batch metrics API.
For ref: https://github.com/celian-garcia/opentelemetry-collector-contrib/blob/50ac164dc41074b64727deb16b7a207461b54508/receiver/azuremonitorreceiver/scraper_batch.go

We followed this doc https://github.com/Azure/azure-sdk-for-go/blob/main/sdk%2Fmonitor%2Fquery%2FMIGRATION.md

I was more asking for the dataplane API but I'm realising, reading again the doc that if we migrate to github.com/Azure/azure-sdk-for-go/sdk/monitor/query/azmetrics for everything, we should be good. Am I right?

[ashishbans]

Hey @celian-garcia , yes, we are good and on right path,

FYI : also mentioned doc you shared
azmetrics contains a new data plane metrics feature that wasn't previously available in azquery. This allows users to query data from one or multiple resource IDs using a data plane endpoint. azmetrics will expand in the future when the service adds more features.

[ashishbans]

Also would you mind sharing the repo fork or pull request where this work is in progress. we also like to contribute to speedup the rollout

[celian-garcia]

Okay. Currently I want to finish this #37167 (multi subscription support) before to rebase and propose a PR from my fork's branch about getBatch

I believe that it's the same for the azmetrics usage. I'd be glad to have your support on this! Can you just wait for the multi subscription config to be merged before starting to migrate to azmetrics. As otherwise we'd have some conflcts in tests.
Also you're more than welcome to review the code that we've done on both multi subscription and getBatch.

[celian-garcia]

/label never-stale -needs-triage good-first-issue