Azure eventhub receiver & elasticsearch exporter- Field [attributes.azure.properties.timestamp] of type [flattened] doesn't support formats.

[vinaydhegde]

Component(s)

No response

What happened?

Description

When OTEL collector is configured with eventhub receiver & elasticsearchexporter,
Elasticsearch exporter is failing to export the logs to elasticsearch with the following errors.

error   elasticsearchexporter@v0.128.0/bulkindexer.go:379       failed to index document        {"resource": {"service.instance.id": "<service_instance_id>", "service.name": "otelcol-contrib", "service.version": "0.128.0"}, "otelcol.component.id": "elasticsearch/otel", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "index": ".ds-<eventhub_dataset>.otel-<datastream_namespace>-<index_id>", "error.type": "document_parsing_exception", "error.reason": ""}

Elasticsearch has the following errors:

error fetching [attributes.azure.properties.timestamp]: Field [attributes.azure.properties.timestamp] of type [flattened] doesn't support formats.
Caused by type - illegal_argument_exception

NOTE:
It was working fine with otelcol-contrib version 0.114.0. May be the timestamp was not available in the attributes.azure.properties.log field, but the field was getting ingested.
issue#36861 - Probably after this enhancement (which was added in 0.117.0), the error started occurring.

Steps to Reproduce

Use the latest version otel contrib collector image (version 0.128.0) with azure eventhub receiver and elasticsearch exporter. See the config in OpenTelemetry Collector configuration block below

Expected Result

Elasticsearch exporter should ingest a multi-field attributes.azure.properties.log with timestamp

Actual Result

No document is written to the index .ds-<eventhub_dataset>.otel-<datastream_namespace>-<index_id>

Collector version

v0.128.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  azureeventhub:
    connection: Endpoint=<eventhub_endpoint>
    format: azure
exporter:
  elasticsearch/otel:
    endpoints:
    - ${env:ELASTIC_ENDPOINT}
    api_key: ${env:ELASTIC_API_KEY}
    logs_dynamic_index:
      enabled: true
    mapping:
      mode: otel

Log output

Additional context

No response

[VihasMakwana]

@vinaydhegde I think you need to change mapping type of that particular field in your Elasticsearch. I suspect this has nothing to do with exporters.

I'll allow @carsonip, @lahsivjar to share their thoughts on this.

[github-actions]

Pinging code owners for exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[carsonip]

@vinaydhegde May I know your Elasticsearch version?

ES 8.16+ comes with mappings for otel data streams that fits index pattern logs-*.otel-*, traces-*.otel-* and metrics-*.otel-*, and they are designed to work with elasticsearch exporter otel mode, which you are using.