[receiver/filelog] Support reading most recent file (by mtime) only

[BinaryFissionGames]

Component(s)

receiver/filelog

Is your feature request related to a problem? Please describe.

We've had a few users running the filelog receiver in environments where logs don't really "roll" traditionally.

Instead, these users are running in environments where a file is written to until some threshold (time, size), then a completely new file is created and written to instead. In these environments, the old files won't be cleaned up for a long time, and also won't be written to again.

When many old files exist in a directory, we experience high CPU utilization. It seems like we could reduce this utilization if we somehow didn't read the file and compare it's fingerprint every cycle, but instead only payed attention to files that grew in size, or had it's date modified.

In benchmarks, I've seen a 25% increase in performance by stat'ing files for mtime and size over reading a fingerprint.

Describe the solution you'd like

There are a couple of solutions that have been discussed:

1. Tap into the sorting logic to allow using the modtime directly on the file, instead of only a regex.
2. Allow fingerprinting logic to have configurable implementations, such that the current behavior (first N bytes) is used by default, but optionally you would be able to use a different implementation that allowed for filename to be used, and only read the file when the size of the file has changed.

Describe alternatives you've considered

The existing sorting logic works here for the most part, but it is:

1. Difficult for users to configure (requires knowledge of filename pattern, specifying a regex, then specifying a date format)
2. Not possible to use in instances where the timestamp is not in the file name

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Allow fingerprinting logic to have configurable implementations, such that the current behavior (first N bytes) is used by default, but optionally you would be able to use a different implementation that allowed for filename to be used, and only read the file when the size of the file has changed.

I think the idea of having alternate implementations of "how to keep track of files" has merit. The current implementation makes minimal assumptions about file rotation, but is not necessarily the most performant in every situation. As you noted, files which sit forever and never grow equate to overhead in every poll cycle.

An alternate implementation which assumes files are never moved or truncated could rely on file size and/or last modified time to track which files have new content. This would be more performant but would fail if the assumptions are violated. I think there could be other implementations too which each have their own advantages and limitations.

Unfortunately, I don't think the current fileconsumer package is quite ready to extract the interface necessary to support this. I am currently refactoring the package into more manageable and testable parts, so hopefully this could be done before long.

In the meantime I'd be interested to see any ideas on how exactly we could isolate file tracking logic. Some things we'd have to address:

• Are there unique settings for each alternate implementation? How do we manage these in the build process?
• What is the "outer" interface which the top-level of package works with?
• Is there an "inner" interface or stripped down version of the "reader", which allows us to share common functionality while avoiding overhead that is specific to one implementation?

---

Tap into the sorting logic to allow using the modtime directly on the file, instead of only a regex.

While this isn't ideal, I do think it's a reasonable way to offer a performance gain in the nearer term. It probably makes sense to keep this behind a feature gate until we can determine for sure whether the file management interface is possible. I think it'll be necessary to implement #23788 as well in order to make this useful.

[BinaryFissionGames]

I'll start by implementing #23788 - seems straightforward enough.

I'll circle back to these questions and see if I have any ideas.

[crobert-1]

Removing needs triage on the basis that this is a valid enhancement request, but still needs some investigation into implementation details.

[cwegener]

When many old files exist in a directory, we experience high CPU utilization

This seems to be the same issue or at least significantly similar to the one that I raised. #27404 Although "high" CPU utilization is open to interpretation.

In the issue I raised, the very rough observation was:

• If number of files increases by 10x (100 -> 1000), CPU utilization increases by 20x (5% -> 110%).

Also, 1000 files might not meet the minimum criteria for "many files" for some people.

I'll run the tests I documented in #27404 against #28691 and see how it changes the results.