Description
Component(s)
exporter/awss3
Is your feature request related to a problem? Please describe.
In the Amazon Cloudwatch Exporter there is a configuration setting called raw_log
which will take the OpenTelemetry format and significantly reduce the size of the JSON and do some processing to make logs easily queryable in CloudWatch logs. I'd love to see this be added to the S3 Exporter. When trying to then do an ingestion of the Logs in S3 into a service like OpenSearch the data in S3 is very hard to parse and process leading to pains when trying to query it. If the feature of raw_log
was enabled for logs exported to S3 this would reduce the burden of engineers having to do transformations in Lambda, ETL jobs or OSI Pipelines.
Describe the solution you'd like
I would want the S3 Exporter to have a raw_log
attribute defaulted to false
that when enabled would emit the same logic the CloudWatch Exporter has to turn the logs into a unified struct before doing a PutObject
to the JSON in S3. This should not impact anything on metrics or traces and should only impact logs.
Describe alternatives you've considered
There is ETL changes after the data is in S3 or some work that can be done via operators in the log receiver used that could be done in place of this however this solution would be much simpler do in the Exporter.
Additional context
The code the Cloudwatch Log Exporter does when raw_log
is set to true lives here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awscloudwatchlogsexporter/exporter.go#L208-L239