Skip to content

prometheusremotewrite emits noisy errors on empty data points #4972

Closed as not planned
@nicks

Description

@nicks

Describe the bug
Here's the error message:

2020-12-01T00:03:54.421Z	error	exporterhelper/queued_retry.go:226	Exporting failed. The error is not retryable. Dropping data.	{"component_kind": "exporter", "component_type": "prometheusremotewrite", "component_name": "prometheusremotewrite", "error": "Permanent error: [Permanent error: nil data point. image_build_count is dropped; Permanent error: nil data point. image_build_duration_dist is dropped]", "dropped_items": 2}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
	/home/circleci/project/exporter/exporterhelper/queued_retry.go:226
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
	/home/circleci/project/exporter/exporterhelper/metricshelper.go:115
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
	/home/circleci/project/exporter/exporterhelper/queued_retry.go:128
github.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1
	/home/circleci/go/pkg/mod/github.com/jaegertracing/[email protected]/pkg/queue/bounded_queue.go:77

Here's the metric descriptor emitted by logging exporter for the same metric:

Metric #0
Descriptor:
     -> Name: image_build_count
     -> Description: Image build count
     -> Unit: ms
     -> DataType: IntSum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE

I don't know enough about the contracts here to know if this is a bug in the opencensus code that I'm using to send the metric, or in the batcher, or in the prometheusremotewrite exporter, or something else entirely.

What did you expect to see?
No error messages

What did you see instead?
An error message

What version did you use?
Docker image: otel/opentelemetry-collector:0.15.0

What config did you use?

    extensions:
      health_check:
      pprof:
        endpoint: 0.0.0.0:1777
      zpages:
        endpoint: 0.0.0.0:55679
    
    receivers:
      opencensus:
        endpoint: "0.0.0.0:55678"
    
    processors:
      memory_limiter:
        check_interval: 5s
        limit_mib: 4000
        spike_limit_mib: 500
      batch:
    
    exporters:
      logging:
        loglevel: debug
      prometheusremotewrite:
        endpoint: "http://.../api/v1/prom/write?db=tilt"
        insecure: true
    
    service:
      extensions: [health_check, pprof, zpages]
      pipelines:
        metrics:
          receivers: [opencensus]
          exporters: [logging, prometheusremotewrite]

Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions