kafkareceiver: extract Kafka message headers as client metadata

[axw]

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

I would like to propagate client metadata (e.g. a tenant ID) all the way to our backend. We have Kafka in the middle of our pipeline, like this:

flowchart LR
  otlpreceiver --> kafkaexporter
  kafkaexporter --> Kafka
  Kafka --> kafkareceiver
  kafkareceiver --> otlpexporter
  otlpexporter --> backend

Loading

otlpreceiver has include_metadata configuration which instructs it to record gPRC metadata (HTTP headers) as client metadata. This can be injected into out going requests using https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetterextension

Neither kafkareceiver nor kafkaexporter currently propagate client metadata.

Describe the solution you'd like

We should add include_metadata configuration to kafkareceiver to extract message headers and record them as client metadata.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy @axw

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[marclop]

Since I'm looking at the kafkaexporter issue, I can work on this one as well.

[crobert-1]

Issue filed by code owner, removing needs triage