Collector memory increases by about ~20 MB after v0.125.0 release

[sfc-gh-bnandibhatla]

Component(s)

service

What happened?

Describe the bug

We noticed that the collector memory increases by about ~20 MB after the latest release. Here's a graph (plotted using otelcol_process_memory_rss metric) showing the memory of the collector running an older version (v0.120.0) and the newer version:

Steps to reproduce

Upgrade the collector to the latest version

What did you expect to see?

Very minimal increase in memory used by the collector.

What did you see instead?

The memory increased by about ~20 MB.

Collector version

v0.125.0

Environment information

Environment

OS: Centos7
Compiler(if manually compiled): go 1.23.7

Additional context

We collected heap profiles from both old and new versions of the collector. Here's the heap profile from old version:

Here's the heap profile from the new version of the collector:

It looks like the memory increase is because of a new usage of zap somewhere (go.uber.org/zap/zapcore.newCounters). Drilling down using the flamegraph shows the call stack:

The call to the zapcore.NewSamplerWithOptions seems to be added here: #12617. The ~20 MB increase seems excessive just to maintains some logging related counters.

[bogdandrutu]

#13015 only fixes the case where OTLP is not used.

[jade-guiton-dd]

I've filed an issue in the Zap repo to ask if it would be possible to share a sampler core between multiple Zap pipelines (one for each OTel Logger we create), which would be one way to eliminate this issue: uber-go/zap#1498

[jade-guiton-dd]

There has been no response to the issue on the Zap repo so far, but I had an idea on how to share sampling counters using reflect; see this draft PR: #13107. It's not super elegant and could break if a new Zap version changes the sampler internals, but it would solve the memory issue in cases where logs are exported through OTLP. If you have context, please take a look and tell me if you think the hack is worth it and I should submit it for review.