Skip to content

[exporter/awss3] Raw Log Setting #32679

Closed
@blakeromano

Description

@blakeromano

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

In the Amazon Cloudwatch Exporter there is a configuration setting called raw_log which will take the OpenTelemetry format and significantly reduce the size of the JSON and do some processing to make logs easily queryable in CloudWatch logs. I'd love to see this be added to the S3 Exporter. When trying to then do an ingestion of the Logs in S3 into a service like OpenSearch the data in S3 is very hard to parse and process leading to pains when trying to query it. If the feature of raw_log was enabled for logs exported to S3 this would reduce the burden of engineers having to do transformations in Lambda, ETL jobs or OSI Pipelines.

Describe the solution you'd like

I would want the S3 Exporter to have a raw_log attribute defaulted to false that when enabled would emit the same logic the CloudWatch Exporter has to turn the logs into a unified struct before doing a PutObject to the JSON in S3. This should not impact anything on metrics or traces and should only impact logs.

Describe alternatives you've considered

There is ETL changes after the data is in S3 or some work that can be done via operators in the log receiver used that could be done in place of this however this solution would be much simpler do in the Exporter.

Additional context

The code the Cloudwatch Log Exporter does when raw_log is set to true lives here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awscloudwatchlogsexporter/exporter.go#L208-L239

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions