Prometheus receiver intermittently dropping kubelet metrics

[alclark704]

Component(s)

exporter/promethuesremotewrite

What happened?

Description

We're using the Prometheus Receiver as a drop in replacement for Prometheus and it's mostly working as expected but we're currently having issues with gaps in the collector kubelet metrics. We have a pool of collectors and Target Allocator deployment, all managed by an OpentelemetryCollector resource by the Opentelemetry Operator, with the collectors writing to an upstream Thanos receive stack. We provide the same static config to the collectors that we provide to Prometheus (which we still currently have running alongside) and what we see is intermittent gaps in the metrics coming from kubelet, just one or 2 dropped scrapes but they cause gaps in the series that skew all of our node based metrics.

There's been some discussion on the CNCF slack so far: https://cloud-native.slack.com/archives/C01LSCJBXDZ/p1748621317097949

But so far we have investigated:

• The collectors aren't under any resource strain when dropping series
• The drops are only happening in kubernetes-nodes and kubernetes-nodes-cadvisor jobs
• The drops are not consistent across all nodes a particular collector is scraping (i.e. all nodes from a particular collector experience it but not at the same time)
• The drops don't correlate to collectors restarting or scaling or nodes starting/stopping
• There's nothing obvious in the collector logs, even with detailed logging (some timeouts to the apiserver but those seem to relate to nodes starting/stopping rather than these gaps)
• Prometheus still running in the same cluster with the same config does not have the same problem

I'm not entirely sure the issue is with the prometheus receiver, given it uses the same scrape logic as Prometheus itself I see no reason why it would have any issue. It could well be differences in the way the collector/exporter manage the series, particularly the kubelet ones with honor_timestamps: true but struggling to narrow down what exactly is happening.

Steps to Reproduce

Otel collector with the below static config, with prometheus remotewrite exporter to Thanos receive stack.

Expected Result

This is from Prometheus in the same cluster monitoring up{job="kubernetes-nodes"} for a given instance:

Actual Result

This is the same instance via the collector:

Collector version

0.122.1

Environment information

Environment

EKS: v1.31.9-eks-5d4a308

OpenTelemetry Collector configuration

exporters:
      debug: {}
      prometheusremotewrite:
        endpoint: ${THANOS_RECEIVER}
        remote_write_queue:
          enabled: true
          num_consumers: 5
        target_info:
          enabled: false
        timeout: 10s
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
    receivers:
      prometheus:
        api_server:
          enabled: true
          server_config:
            endpoint: localhost:9090
        config:
          global:
            scrape_interval: 15s
          scrape_configs:
          - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            honor_timestamps: true
            job_name: kubernetes-nodes
            kubernetes_sd_configs:
            - role: node
            relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_node_label_(.+)
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$$1/proxy/metrics
              source_labels:
              - __meta_kubernetes_node_name
              target_label: __metrics_path__
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
          - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            honor_timestamps: true
            job_name: kubernetes-nodes-cadvisor
            kubernetes_sd_configs:
            - role: node
            metric_relabel_configs:
            - action: drop
              regex: container_cpu_(load_average_10s|system_seconds_total|user_seconds_total)
              source_labels:
              - __name__
            - action: drop
              regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
              source_labels:
              - __name__
            - action: drop
              regex: container_memory_(mapped_file|swap)
              source_labels:
              - __name__
            - action: drop
              regex: container_(tasks_state|threads_max)
              source_labels:
              - __name__
            - action: drop
              regex: container_spec_(cpu.*|memory_swap_limit_bytes|memory_reservation_limit_bytes)
              source_labels:
              - __name__
            - action: drop
              regex: .+;
              source_labels:
              - id
              - pod
            relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_node_label_(.+)
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$$1/proxy/metrics/cadvisor
              source_labels:
              - __meta_kubernetes_node_name
              target_label: __metrics_path__
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
          - job_name: kubernetes-service-endpoints
            kubernetes_sd_configs:
            - role: endpoints
            relabel_configs:
...
          - job_name: kubernetes-pods
            kubernetes_sd_configs:
            - role: pod
            relabel_configs:
...
        target_allocator:
          collector_id: ${POD_NAME}
          endpoint: http://opentelemetry-metrics-targetallocator
          interval: 30s
    service:
      pipelines:
        metrics:
          exporters:
          - prometheusremotewrite
          - debug
          processors:
          - memory_limiter
          receivers:
          - prometheus

Log output

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole @ArthurSens @krajorama

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@alclark704 Hello!

Thanks for detailed information.
I have not worked with prometheus before but does your collector instance throw any error? You can probably enable debug logging as well.

[alclark704]

Hi @VihasMakwana

Unfortunately, no errors or info that seems relevant around the time of the gaps via the debug exporter. I've also gone up to the detailed logging and so far not found anything useful in there though with multiple lines for every scrape it's quite hard to sift through.

[dashpole]

Are you able to tell which collector is scraping which node at any point in time (e.g. by adding a constant label from each collector)? It would be helpful to know if that is happening when the target is being moved between collectors, or when it is assigned to a single one.

/api/v1/nodes/$$1/proxy/metrics

Note that this means you are proxying metrics through your kubernetes APIserver. Any APIServer timeout would mean you don't have metrics. Generally, I would recommend using a daemonset to directly scrape the kubelet, rather than sharding using the targetallocator and scraping by using the APIServer as a proxy.

Can you check the up metric for each target, and look at the scrape_* metrics the Prometheus receiver adds automatically? Those can help show whether this is because of failed scrapes or something else.

[alclark704]

Hi @dashpole,

Thanks for the response! I can grab some dashboards if it helps but this is happening when a target is consistently on a single collector. We did have the same thought which was the rationale behind checking if the collectors were scaling up/down when the gaps occurred but that wasn't the case. I think with the consistent-hashing we shouldn't see targets pass between collectors unless the number of collectors changes (?). We do occasionally get duplicated targets where one is passed between 2 collectors but that's much less frequent and gets de-duplicated by Thanos.

Absolutely agree it's not the way we want to do the node metrics in particular long term. With the migration to otel we're trying to minimise changing too many things at once so were hoping to keep the config itself the same and just verify we can recreate the current stack with Otel for now but do want to change that in the future. Though it might be worth moving that up if this persists. And have also considered the API server timeouts, I checked the scrape duration on the metrics in question and found the duration was perfectly normal either side of the gap but the scrape duration metric itself is also missing in those gaps. And found it odd that I think under the hood Prometheus and the Collector should both be hitting the same endpoint with the same scrape logic but 1 consistently sees timeouts and the other doesn't see any, if it was due to the API server timing out. I did start digging through apiserver logs to see if there was anything in there but didn't go too far so maybe worth going back to that.

I think I've covered a bit of the last point but yeah the screenshots attached above are the up{} metrics for a given node and the scrape_* metrics are all also missing for the same node in the same gaps, like this:

Very much appreciate the suggestions and if there's anything else you can think of to check I'm happy to take a look.

[dashpole]

If the up metric is missing during the window then that almost certainly indicates that the problem is happening after the Prometheus receiver. It emits up=0 when a scrape fails, so the metric should always be present even when scrapes fail.

[alclark704]

Ok, that is good to know, thank you very much for your help. I guess next port of call is the remote write exporter so have just tried updating the component in the original body to that but not sure if that'll actually relabel the issue