Filelog receiver does not support mixed compression use case

[verejoel]

Component(s)

receiver/filelog

What happened?

Description

A common use-case is to read a live uncompressed log, which is typically rotated on a daily basis into a compressed version with some date suffix.

Now, to support outages of the collector (e.g. let's say the service was stopped for 3 days) we'd like to have a wildcard entry on the include statement to pick up all the logs, and backfill them across the duration of the outage.

As far as I can tell, it is not currently possible to support this use-case in the filelog receiver. We could potentially support the use case with two separate filelog receivers, however, this is actually a bad approach as every time the file is rotated, the entire batch of logs will be shipped again by the second scraper.

Steps to Reproduce

filelog/frr:
    # wildcard to scoop up raw log `test.log` and rotated logs `test.log.0.gz
    include: [ "/var/logs/test.log*" ]
    start_at: beginning
    # persistent storage for positions file
    storage: file_storage/frr

Setup a receiver like this, and observe how it can happily ingest test.logs. But when we rotate (mv to test.logs.0, then gzip) the new log is shipped, but with binary content in the string. So it is recognized as a new file and shipped twice (but with gzipped data).

Setting compression: gzip results in the raw log not being ingester, and the following errors:

{"level":"error","ts":"2025-02-07T14:50:45.977+0100","caller":"reader/reader.go:82","msg":"failed to create gzip reader","service.name":"otelcol-osag","kind":"receiver","name":"filelog/test","data_type":"logs","component":"fileconsumer","path":"/tmp/test.log","error":"gzip: invalid header"}

Expected Result

I would expect this use-case to be supported. When the log is rotated and compressed, the compressed data should not be shipped again.

Actual Result

We cannot watch compressed and uncompressed logs at the same time.

Collector version

0.118.0

Environment information

Environment

Running on K8s. 1.30

OpenTelemetry Collector configuration

Log output

Additional context

This could probably be solved by inspecting the file MIME type, and inspecting the uncompressed fingerprint to determine if the file has been seen before.

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Can you use two separate instances of the receiver?

filelog/raw:
    include: [ "/var/logs/test.log" ]
    ...
filelog/gzip:
    include: [ "/var/logs/test.log.*.gz" ]
    compression: gzip
    ...

[verejoel]

Can you use two separate instances of the receiver?

Iam pretty sure not. As when the log is rotated (so it no longer matches filelog/raw, it will be picked up by filelog/gzip and the data will be shipped again.

[djaglowski]

Sorry, I see that I skimmed your original comment and missed the answer to my question.

It seems there are two parts to solving this.

1. Fingerprinting of compressed files would have to be changed such that we fingerprint them based on their decompressed content.
2. Assuming (1) is possible, the user should be able to explicitly indicate the file formats they'd like to support. I think this is necessary because some users may want to pick up compressed files in whole and won't want them automatically decompressed. Since we may add support for additional compression types in the future, we may need to change compression into a slice.

Overall this change makes sense to me. I'd be happy to review PRs if anyone has time to try an implementation.

[VihasMakwana]

@djaglowski How would we know a file that a file is compressed? I suppose we can determine that based on their headers. For eg. gzip header starts with magic number 1f 8b. We should initially support common compression methods.

Overall, I support this enhancement and I think it will be a valuable addition.

[djaglowski]

How would we know a file that a file is compressed?

I think we'd need to document our assumptions. Inferring from the file name extension seems like a decent starting point though.

[VihasMakwana]

@djaglowski yeah, let's aim to keep things simple.

[khushijain21]

can you assign this issue to me?