Filtering metrics by label/attribute (via regex)

[LilWatson]

Component(s)

processor/filter

What happened?

Description

My original goal was to include only metrics with the label namespace starting with a given string. Since this did not work i tried filtering with a static string, which also gave me problems.

Steps to Reproduce

kind create cluster
install mimir (chart default values)
install kube-state-metrics exporter (chart default values)

# kustomize.yaml with 
helmCharts:
# https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector
- name: opentelemetry-collector
  releaseName: tenant
  valuesInline:
    mode: deployment
    image:
      repository: otel/opentelemetry-collector-contrib
    config:
      extensions:
        health_check:
          endpoint: ${env:MY_POD_IP}:13133
      receivers:
        prometheus:
          config:
            scrape_configs:
              - job_name: opentelemetry-collector
                scrape_interval: 10s
                static_configs:
                  - targets:
                    # ClusterIP of kube-state-metrics
                    - 10.96.87.132:8080
      processors:
        # Based on https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.96.0/processor/filterprocessor/README.md#alternative-config-options
        filter:
          metrics:
            include:
              match_type: regexp
              resource_attributes:
                - key: namespace
                  value: graf.*
      exporters:
        debug:
          verbosity: normal
        prometheusremotewrite:
          endpoint: http://mimir-nginx.mimir.svc:80/api/v1/push
          headers:
            X-Scope-OrgID: my-project
      service:
        extensions:
          - health_check
        pipelines:
          metrics:
            receivers: [prometheus]
            processors: [filter]
            exporters: [prometheusremotewrite, debug]
  version: 0.108.0
  repo: https://open-telemetry.github.io/opentelemetry-helm-charts

Expected Result

The original config should include only metrics/timeseries with a namespace label starting with "graf".
The additional filters i have tried (see configuration section) should exclude timeseries with a namespace label set to "grafana".

Actual Result

The original config result also includes metrics/timeseries with a namespace label not starting with "graf".
The additional filters result includes timeseries with a namespace label set to "grafana".

Collector version

v0.111.0

Environment information

Environment

OS: macOS 14.0 (23A344)

OpenTelemetry Collector configuration

extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: opentelemetry-collector
          scrape_interval: 10s
          static_configs:
            - targets:
              - 10.96.87.132:8080
processors:
  # Based on https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.96.0/processor/filterprocessor/README.md#alternative-config-options
  filter:
    metrics:
      include:
        match_type: regexp
        resource_attributes:
          - key: namespace
            value: graf.*
    exporters:
  debug:
    verbosity: normal
  prometheusremotewrite:
    endpoint: http://mimir-nginx.mimir.svc:80/api/v1/push
    headers:
      X-Scope-OrgID: my-project
service:
  extensions:
    - health_check
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [filter]
      exporters: [prometheusremotewrite, debug]

# Additional filter processors

# Based on https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md#examples
filter:
  metrics:
    datapoint:
    - 'resource.attributes["namespace"] == "grafana"'

# Based on https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md#hasattrondatapoint
filter:
  metrics:
    metric:
      - 'HasAttrOnDatapoint("namespace", "grafana")'

# Based on https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.96.0/processor/filterprocessor/README.md#using-an-expr-match_type
filter:
  metrics:
    exclude:
      match_type: expr
        expressions:
        - Label("namespace") == "grafana"

Log output

# Original config
k logs -l "app.kubernetes.io/instance=tenant" -n otel-col | grep -v namespace=grafana
kube_daemonset_status_number_available{daemonset=kube-proxy,namespace=kube-system} 1

# Additional filter processors
k logs -l "app.kubernetes.io/instance=tenant" -n otel-col | grep namespace=grafana
kube_pod_container_status_running{container=grafana,namespace=grafana,pod=grafana-684d8b87db-7zklg,uid=bf6ffb23-746b-476b-8e95-fd6a26da572a} 1

Additional context

The Include/Exclude filter config featuring regex examples was removed from the readme. While using it there was no configuration error nor warning message. Is this still supported? If not what is the current way to filter metric labels/attributes via regex?

Thanks in Advance! :)

[github-actions]

Pinging code owners:

• processor/filter: @TylerHelmuth @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

Hi @LilWatson! I'm currently trying to reproduce this behaviour, however it seems like the regex filter is correctly applied in my case (using the same configuration). May I ask you to also post the payload of the prometheus endpoint, i.e. of the kube-stats-metrics prometheus endpoint - either the full payload or an example of a metric that should be included, versus one that should not be included?

Regarding the support of the regexp match type: These config options are apparently still supported (I checked the code and they are still included in the config structs and considered in the filtering logic), but according to the docs https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.96.0/processor/filterprocessor/README.md#alternative-config-options these will be deprecated and the recommended way will be to use OTTL expressions - for this use case, the IsMatch function might be suited - I will try to put together a config example for this :)

[bacherfl]

So here would be the filter configuration using the IsMatch() function:

  filter:
    metrics:
      metric:
        - 'IsMatch(resource.attributes["namespace"], "graf.*") != true'

This should exclude all resource metrics where the namespace attribute does not start with graf