Description
Component(s)
processor/cumulativetodelta
Describe the issue you're reporting
We have a usecase, we receive data from our producers on our otel, then we export the data to dynatrace.
To mention, we use 'otlphttp/export-to-dynatrace' exporter for exporting it to dynatrace.
We are receiving below warnings in our otel logs and can also see some data is being dropped.
2024-06-20T12:17:11.450Z warn [email protected]/otlp.go:358 Partial success response {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/export-to-dynatrace", "message": "The following issues were encountered while ingesting OTLP metrics:\nErrors:\nUnsupported metric: 'traces_request_total' - Reason: UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM\n", "dropped_data_points": 58}
2024-06-20T12:17:11.680Z warn [email protected]/otlp.go:358 Partial success response {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/export-to-dynatrace", "message": "The following issues were encountered while ingesting OTLP metrics:\nErrors:\nUnsupported metric: 'system.disk.io' - Reason: UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM\n", "dropped_data_points": 318}
2024-06-20T12:17:11.749Z warn [email protected]/otlp.go:358 Partial success response {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/export-to-dynatrace", "message": "The following issues were encountered while ingesting OTLP metrics:\nErrors:\nUnsupported metric: 'prometheus_tsdb_total' - Reason: UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM\n", "dropped_data_points": 541}
Then we got to know about the "cumulativetodelta" processor.
We quickly tried it in our use case with the below config. After configuring it in the pipeline, the above errors are gone.
processors:
cumulativetodelta/export-to-dynatrace:
But currently with the above configurations, the otel will utilise its resources on every metric for cumulativetodelta processor.
And in our case we are receiving the above mentioned error/warning "UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM" for a large set of metrics.
It does not looks good to add every metrics in "include" parameter of the processor nor do they have a common string so regex can be configured. If in future some metrics are added by consumers and we receive same errors then we have to manually keep adding those metrics in the "include" parameter which is again a headache.
Question is: Do we have some way to include metrics based upon the type? Like we do in "filterprocessor" example below.
processors:
filter/ottl:
metrics:
metric:
- 'type == METRIC_DATA_TYPE_HISTOGRAM'
If we have some way to "include" metrics based on types apart from string & regex, it would be great.
Any other production solution is also appreciated for the usecase by the experts.