Changes in PR #11093 Broke Disabling Internal Telemety For OSes On Which It Isn't Implemented (ie. AIX)

[Dylan-M]

Describe the bug
Previously, I had the contrib collector running on AIX. However, after #11093, specifically this commit, this is now broken with the following error:

# ./collector_aix_ppc64 --config config.yaml
2025-01-14T13:20:29.921-0600    info    service@v0.116.0/service.go:164 Setting up own telemetry...
2025-01-14T13:20:29.921-0600    info    service@v0.116.0/service.go:207 Skipped telemetry setup.
Error: failed to register process metrics: not implemented yet
2025/01/14 13:20:29 collector server run finished with error: failed to register process metrics: not implemented yet

This is because the gopsutil library doesn't yet support AIX for these metrics.

I have this in my config, which should disable the internal telemetry, and as noted above previously worked:

    telemetry:
        metrics:
            level: none

Steps to reproduce
Build contrib for AIX (GOOS=AIX GOARCH=ppc64), scp binary to a suitable (I have 7.3 TL3) AIX system, and run it with the config below.

What did you expect to see?
A running collector.

What did you see instead?
Collector attempts to start, and this logged to stdout:

# ./collector_aix_ppc64 --config config.yaml
2025-01-14T13:20:29.921-0600    info    service@v0.116.0/service.go:164 Setting up own telemetry...
2025-01-14T13:20:29.921-0600    info    service@v0.116.0/service.go:207 Skipped telemetry setup.
Error: failed to register process metrics: not implemented yet
2025/01/14 13:20:29 collector server run finished with error: failed to register process metrics: not implemented yet

What version did you use?
v0.116.0

What config did you use?

receivers:
    filelog/clcomd:
        encoding: utf-8
        fingerprint_size: 1kb
        force_flush_period: 500ms
        include:
            - /var/log/clcomd/*
        include_file_name: true
        include_file_name_resolved: false
        include_file_path: false
        include_file_path_resolved: false
        max_concurrent_files: 1024
        max_log_size: 1MiB
        operators:
            - field: attributes.log_type
              type: add
              value: file
        poll_interval: 200ms
        retry_on_failure:
            enabled: false
        start_at: beginning
processors:
exporters:
    nop/BlackHole: null
service:
    pipelines:
        logs:
            receivers:
                - filelog/clcomd
            exporters:
                - nop/BlackHole
    telemetry:
        metrics:
            level: none

Environment
OS: AIX 7.3 TL3 Standard w/yum,dnf,xlc (can rented from SiteOx, if anyone needs to be able to rent a system for testing purposes)
Compiler: go version go1.23.4 linux/amd64
Using Collector Builder 0.116.0 cross building under Fedora Linux

Additional context
It has been an ongoing struggle to keep the collector working on AIX (and other UNIXes) with lack of due consideration on how changes might affect things when they're "not implemented" in the libraries for a given OS. This is a critical path for us. This previously happened with a change around using Flock(), for example.

I do plan on adding this capability to the gopsutil library so that it will be available in the future. I've been slowly working through enriching the AIX support in that library.

[mx-psi]

@Dylan-M Thanks for your issue. From the platform support doc:

For platforms not listed as supported by any of the tiers, support cannot be assumed to be provided. While the project may accept specific changes related to these platforms, there will be no official builds, support of issues and development of enhancements or bug fixes for these platforms. Future development of project for supported platforms may break the functionality of unsupported platforms.

Support for AIX has come up before, I would be willing to consider Tier 3 support (guaranteed to build) if you want to sign up as a platform owner. It sounds like you are already doing this work, so doing it in a more 'official' capacity wouldn't be a big change and having CI tests would make your life a bit easier.

[Dylan-M]

@Dylan-M Thanks for your issue. From the platform support doc:

For platforms not listed as supported by any of the tiers, support cannot be assumed to be provided. While the project may accept specific changes related to these platforms, there will be no official builds, support of issues and development of enhancements or bug fixes for these platforms. Future development of project for supported platforms may break the functionality of unsupported platforms.

Support for AIX has come up before, I would be willing to consider Tier 3 support (guaranteed to build) if you want to sign up as a platform owner. It sounds like you are already doing this work, so doing it in a more 'official' capacity wouldn't be a big change and having CI tests would make your life a bit easier.

Thanks for responding. I'm not sure my Golang skills are good enough. I'm only working on this for my employer because I have the AIX domain knowledge. Of course, perhaps the AIX domain knowledge is what is needed here. I'd be happy to help in whatever fashion fits within my skill set.

We're also looking at needing to support FreeBSD.

With the wide spread adoption of OTel over the last few years, and the drive to monitor all of our infrastructure, more OS support can only be a good thing.

[mx-psi]

Of course, perhaps the AIX domain knowledge is what is needed here

If you know how to use ocb and cross build, adding CI checks for guaranteeing the Collector builds on the AIX target should not be too hard for you :)

Of course, this issue happens at runtime so it would need a different kind of fix that would require some Go knowledge, but I am sure we can help here.

We're also looking at needing to support FreeBSD.

Can we move the FreeBSD support to a separate issue? I think we have not gotten a lot of asks for this one, so I would like to track it separately.

[Dylan-M]

If you know how to use ocb and cross build, adding CI checks for guaranteeing the Collector builds on the AIX target should not be too hard for you :)

Of course, this issue happens at runtime so it would need a different kind of fix that would require some Go knowledge, but I am sure we can help here.

Yeah. I'm working on the fix to the upstream library this week and next. So, hopefully it won't be an issue for much longer.

Can we move the FreeBSD support to a separate issue? I think we have not gotten a lot of asks for this one, so I would like to track it separately.

Nothing to track yet, the AIX work is the priority for the moment. Was just mentioning it as it relates to UNIX in general is becoming more of a thing with the collector, and it might be a good idea to consider that going forward. This is especially true for us (I work for Bindplane, formerly known as observIQ), as we have many enterprise customers and prospects asking about various UNIX support.

[Dylan-M]

And yes, after a discussion with Dan J., I'm willing to accept the challenge of accepting platform owner for AIX.

[mx-psi]

And yes, after a discussion with Dan J., I'm willing to accept the challenge of accepting platform owner for AIX.

@Dylan-M Awesome! 🥳 Happy to help (over here or on the CNCF Slack) if you want to start adding the tests

Was just mentioning it as it relates to UNIX in general is becoming more of a thing with the collector, and it might be a good idea to consider that going forward. This is especially true for us (I work for Bindplane, formerly known as observIQ), as we have many enterprise customers and prospects asking about various UNIX support.

I think it makes total sense to have FreeBSD as a Tier 3 platform.

[mdirkse]

We ran into the same issue but hacked the collector source code locally before compiling so that we'd still get a working binary. As we also don't need internal metrics this wasn't an issue for us. I looked into adding AIX support for gopsutls, but I don't know nearly enough about AIX to get anywhere. @Dylan-M are you still working on a PR for that?

[Dylan-M]

We ran into the same issue but hacked the collector source code locally before compiling so that we'd still get a working binary. As we also don't need internal metrics this wasn't an issue for us. I looked into adding AIX support for gopsutls, but I don't know nearly enough about AIX to get anywhere. @Dylan-M are you still working on a PR for that?

Affirmative. I expect to have the library updated with AIX support for processes around the end of next week. We have a company hackathon on Thursday the 20th, during which I plan to wrap this up if I can. Otherwise on Friday the 21st.