[receiver/hostmetrics] filesystem scraper doesn't respect root_path

[povilasv]

Component(s)

receiver/hostmetrics

What happened?

Description

Looks like #35504 gave us a regression.

Now filesystem scraper ignores root_path and tries to just open /mounts althought root_path is set. Uses with root_path will start getting these errors, and no filesystem metrics:

2024-10-25T08:45:13.299+0300    error   scraperhelper/scrapercontroller.go:205  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "data_type": "metrics", "error": "open /mounts: no such file or directory", "scraper": "filesystem"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:205
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:177

IMO this is an important regression, as we use this in opentelemetry-helm-charts hostMetrics preset -> https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/templates/_config.tpl#L71-L88

Steps to Reproduce

receivers:
  hostmetrics:
    root_path: /tmp
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.time:
            enabled: true
      disk:
        metrics:
          system.disk.io:
            enabled: true
          system.disk.operations:
            enabled: true
      filesystem:
        metrics:
          system.filesystem.usage:
            enabled: true
      load:
        metrics:
          system.cpu.load_average.1m:
            enabled: true
          system.cpu.load_average.5m:
            enabled: true
          system.cpu.load_average.15m:
            enabled: true
      memory:
        metrics:
          system.memory.usage:
            enabled: true
      network:
        metrics:
          system.network.connections:
            enabled: true
          system.network.io:
            enabled: true
      paging:
        metrics:
          system.paging.operations:
            enabled: true
          system.paging.usage:
            enabled: true
      process:
        metrics:
          process.cpu.time:
            enabled: true
          process.memory.usage:
            enabled: true

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [debug]

  telemetry:
    logs:
      level: "debug"

You will get:

2024-10-25T09:03:56.691+0300    error   scraperhelper/scrapercontroller.go:205  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "data_type": "metrics", "error": "open /mounts: no such file or directory", "scraper": "filesystem"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:205
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:177

although it should try to open /tmp/mounts

Expected Result

Actual Result

Collector version

v0.102.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/hostmetrics: @dmitryax @braydonk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@povilasv is there a work around?

[povilasv]

Workaround is to manually set the env var:

HOST_PROC_MOUNTINFO=/hostfs/proc/1

[marcelaraujo]

@povilasv, there is the same issue here after upgrading to the latest version.

{"level":"error","ts":1729866966.393247,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"open /mounts: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:181"}
{"level":"error","ts":1729866996.3939178,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"open /mounts: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:181"}
{"level":"error","ts":1729867026.3932698,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"open /mounts: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:181"}

[povilasv]

@marcelaraujo try setting the environment variable to HOST_PROC_MOUNTINFO=/hostfs/proc/1

it should workaround this issue

[marcelaraujo]

Hi @povilasv

It didn't work.

What should be the values for this case?

env:
   - name: HOST_PROC_MOUNTINFO
     value: /proc/1/self
volumes:
   - name: hostfs
     hostPath:
        path: /
volumeMounts:
   - name: hostfs
     mountPath: /hostfs
     readOnly: true
     mountPropagation: HostToContainer
config:
   receivers:
      hostmetrics:
          root_path: /hosts

When I tried using your environment variable, I got a different issue

{"level":"error","ts":1729870334.3869734,"caller":"scraperhelper/scrapercontroller.go:205","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"failed to read usage at /hostfs/conf: no such file or directory; failed to read usage at /hostfs/hostfs/run/containerd/io.containerd.runtime.v2.task/k8s.io/b9d436f09506a98c09ba162dc17d2692f70430adf0601dfc1cd2f676c0253b80/rootfs/conf: no such file or directory","scraper":"hostmetrics","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:205\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:177"}

[atoulme]

Looking at this issue and going to attempt to reproduce ; the environment variable is used in 2 places and setting it as a workaround might not be the fix.

[atoulme]

Using your config file, on Mac, trying to reproduce:

docker run --rm -it -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml -v /:/tmp otel/opentelemetry-collector-contrib:latest

I don't see the error reported. I will try to reproduce on a Linux VM next.

[atoulme]

Reproducing on Linux now, taking it further.

[atoulme]

Adding -e HOST_PROC_MOUNTINFO=/tmp/proc/1 doesn't fix the issue.

2024-10-25T16:20:16.489Z	error	scraperhelper/scrapercontroller.go:205	Error scraping metrics	{"kind": "receiver", "name": "hostmetrics", "data_type": "metrics", "error": "failed to read usage at /tmp/tmp/boot: no such file or directory; failed to read usage at /tmp/tmp/boot/efi: no such file or directory; failed to read usage at /tmp/tmp/snap/snapd/21759: no such file or directory; failed to read usage at /tmp/tmp/snap/amazon-ssm-agent/7993: no such file or directory; failed to read usage at /tmp/tmp/snap/core18/2829: no such file or directory; failed to read usage at /tmp/tmp/snap/core22/1621: no such file or directory; failed to read usage at /tmp/tmp/snap/amazon-ssm-agent/9565: no such file or directory; failed to read usage at /tmp/tmp/snap/core18/2846: no such file or directory; failed to read usage at /tmp/tmp/snap/snapd/22991: no such file or directory; failed to read usage at /tmp/tmp/snap/core22/1663: no such file or directory; failed to read usage at /tmp/etc/otelcol-contrib/config.yaml: no such file or directory", "scraper": "hostmetrics"}

[atoulme]

Using -e HOST_PROC_MOUNTINFO="" fixes the issue, setting the correct default value for this env var. The fix is to not set the default value of HOST_PROC_MOUNTINFO in the envMap at all as any non-empty value is going to skew things. I will open a PR in a second with a fix.