Prometheus Remote Write Exporter not sending data for 15min at a time

[obs-gh-mattcotter]

Component(s)

exporter/prometheusremotewrite

Describe the issue you're reporting

We had an ongoing issue with the Prometheus Remote Write Exporter where all metrics would fail to send for 15 minutes at a time and then it would recover on its own. The published stats from the internal telemetry showed otelcol_exporter_sent_metric_points_total drop to 0. The queue size (otelcol_exporter_queue_size) remained at 0, and the http endpoint that the PRWE was configured to hit did not receive any traffic. We enabled debug logging, but the PRWE seems to not write any debug logs. The error message we saw is:

error	internal/queue_sender.go:46	Exporting failed. Dropping data.	{"otelcol.component.id": "prometheusremotewrite/observe", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "error": "Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded", "errorCauses": [{"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}], "dropped_items": 638}
	/home/runner/work/observe-agent/observe-agent/vendor/go.opentelemetry.io/collector/exporter/exporterqueue/async_queue.go:47
go.opentelemetry.io/collector/exporter/exporterqueue.(*asyncQueue[...]).Start.func1
	/home/runner/work/observe-agent/observe-agent/vendor/go.opentelemetry.io/collector/exporter/exporterhelper/internal/batcher/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/batcher.(*disabledBatcher[...]).Consume
	/home/runner/work/observe-agent/observe-agent/vendor/go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue_sender.go:46
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1

This issue was present in v0.118.0, but we upgraded to v0.124.0 and it seems to be fixed. I am hoping to better understand the root cause since it was never listed as a bug fix under the PRWE (I am guessing it may have been related to an exporter helper fix). Can anyone help point me to what the fix might have been? Thank you!

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole @ArthurSens @ywwg

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[obs-gh-mattcotter]

/label help-wanted

[bacherfl]

(Triage) removing needs-triage, and adding waiting-for-code-owners as the issue is well described and contains all relevant information

[obs-gh-mattcotter]

/label bug

Thanks for triaging, I appreciate the help! I checked my data again after the weekend and it seems that this issue is still present in v0.124.0, so I'm adding the Bug label. The errors seem to have gotten less frequent, which is what caused me to assume it had been fixed.

Here is the PRWE config I'm using:

prometheusremotewrite:
  endpoint: "[removed]"
  # headers: removed
  max_batch_request_parallelism: 10
  remote_write_queue:
    num_consumers: 10
  resource_to_telemetry_conversion:
    enabled: true
  send_metadata: true
  timeout: 10s

The collector is running with --feature-gates=+exporter.prometheusremotewritexporter.EnableMultipleWorkers, but we also tried without that flag and omitting the remote_write_queue/max_batch_request_parallelism config, and we hit the same issue.

The PRWE is used in the following two pipelines:

metrics/hostmetrics:
  exporters:
  - prometheusremotewrite
  processors:
  - memory_limiter
  - k8sattributes
  - batch
  - resourcedetection/cloud
  - resource
  - attributes/debug_source_hostmetrics
  receivers:
  - hostmetrics
metrics/kubeletstats:
  exporters:
  - prometheusremotewrite
  processors:
  - memory_limiter
  - k8sattributes
  - batch
  - resourcedetection/cloud
  - resource
  - attributes/debug_source_kubeletstats_metrics
  receivers:
  - kubeletstats

[jmichalek132]

Context deadline exceeded, sound like the http request is hitting a timeout, did you observe any increase latency in the backend receiving the data? Does it come and go? If it happens again can you hit the target endpoint with a curl and observe the latency?

[obs-gh-mattcotter]

The failures do come and go, but they always happen for exactly 15 minutes. The latency on the endpoint isn't noticeably different at these times; the p99 request time I'm seeing is consistently <500ms, which is well within the 5s default timeout (especially when the queue is empty). But during this time, I don't see any evidence of the prometheus http endpoint being hit at all; I updated the user agent header for the PRWE and logged nginx traffic and requests stop showing up during this period. The otlphttp exporter from the same collector is able to successfully hit the same nginx gateway during the PRWE error time, so I don't believe it to be a networking issue.

[jmichalek132]

The failures do come and go, but they always happen for exactly 15 minutes. The latency on the endpoint isn't noticeably different at these times; the p99 request time I'm seeing is consistently <500ms, which is well within the 5s default timeout (especially when the queue is empty). But during this time, I don't see any evidence of the prometheus http endpoint being hit at all; I updated the user agent header for the PRWE and logged nginx traffic and requests stop showing up during this period. The otlphttp exporter from the same collector is able to successfully hit the same nginx gateway during the PRWE error time, so I don't believe it to be a networking issue.

Hmm, does it happened to both pipelines or just one? You don't have wal enabled right?
I assume when this happens you still have data coming into the pipeline right?
Do you see these errors you show in logs for the whole duration, is there any chance the data is stuck in a different part of the pipeline?
Could you potentially add the debug exporter too so we can see the data coming to it?
Could you collect profiling data from the otel instances when this happens?