receiver/prometheus: Enable deactivation of Created Timestamp parsing based on user configuration

[zarbis]

Component(s)

receiver/prometheus

What happened?

Originally reported here: #38904

After upgrading from 0.117.0 to 0.126.0 I've noticed roughly 5-10 times increase in CPU usage.
I've narrowed down regression to be in version 0.121.0.

Some additional info:

• I run logs, metrics and traces collectors as separate OTECOLs via otel-operator
• This issue is present ONLY in metrics collector

Here is CPU profile:

Memory:

Goroutines:

Collector version

0.121.0

Environment information

Environment

EKS 1.31

OpenTelemetry Collector configuration

config:
  connectors:
    forward/all: {}
  exporters:
    debug/detailed:
      sampling_initial: 5
      sampling_thereafter: 100
      verbosity: detailed
    debug/normal:
      verbosity: normal
    otlphttp:
      auth:
        authenticator: basicauth/client
      compression: gzip
      endpoint: <REDACTED>
  extensions:
    basicauth/client:
      client_auth:
        password: ${OTEL_GW_PASSWORD}
        username: ${OTEL_GW_USERNAME}
    health_check:
      endpoint: 0.0.0.0:13133
      path: /
    pprof:
      endpoint: 0.0.0.0:1777
  processors:
    attributes:
      actions:
      - action: upsert
        key: cluster
        value: <REDACTED>
      - action: upsert
        key: alerting_tier
        value: "1"
    batch:
      send_batch_max_size: 5000
      send_batch_size: 2000
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    prometheus:
      config:
        scrape_configs:
        - job_name: dummy
          static_configs:
          - targets:
            - 127.0.0.1:8888
  service:
    extensions:
    - basicauth/client
    - health_check
    - pprof
    pipelines:
      metrics/forward:
        receivers:
        - otlp
        - prometheus
        processors:
        - memory_limiter
        - attributes
        exporters:
        - forward/all
      metrics/otlp:
        receivers:
        - forward/all
        processors:
        - batch
        exporters:
        - otlphttp

Log output

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole @ArthurSens @krajorama

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[gracewehner]

+1 we are also seeing 5-10x more CPU usage on some clusters after upgrading to 0.121.0. The difference between 0.120.0 and 0.121.0 that correlates with the pprof above seems to be this PR: #36660 that enables the created-timestamps-zero-injection feature flag in Prometheus. This now calls the extra function CreatedTimestamp() in each scrape loop: https://github.com/prometheus/prometheus/blob/1d9dfde9894f0c82b809b70aa5f9cecab25b8fea/scrape/scrape.go#L1770-L1799 which is defined here: https://github.com/prometheus/prometheus/blob/main/model/textparse/openmetricsparse.go#L287 with a comment that it may consume additional CPU/Memory resources

[ArthurSens]

Great catch! Thanks for sharing. I assume you both have OpenMetrics enabled, right? Since only the OpenMetrics parser is showing the performance degradation: prometheus/prometheus#14808

I believe it's worth letting users de activate the CreatedTimestamp parsing, since OM has poor performance 🤔

[ArthurSens]

Alright, we have two PRs doing the same thing but with slightly different approaches:

• [receiver/prometheus] add feature flag to be able to disable Prometheus setting to fix high CPU for openmetrics #40355 - Feature flag
• [receiver/prometheusreceiver]: Allow to configure EnableCreatedTimest… #40380 - Config option

Thanks @gracewehner and @LukaszByczynski for tackling this!

We probably want to reach consensus on the approach, though. I'm a bit more inclined to use a configuration option, since feature-flags are meant to reach maturity someday and therefore we'll eventually enable parsing Created Timestamps for everyone. What do you both think? And @dashpole @ridwanmsharif , any opinions?