Load balancing exporter is not forwarding HTTP headers

[pavleec]

Component(s)

exporter/loadbalancing

What happened?

Description

I have been trying to configure load balancing exporter to forward the X-Scope-OrgID header (passed via OTEL_EXPORTER_OTLP_TRACES_HEADERS) to the backend otel collectors but with no success. It's not working w/ and w/o using headers_setter on LB. However it works properly if I connect directly to the backend collector which is exporting traces to multitenant Grafana Tempo. Thanks in advance for any help.

Collector version

0.117.0

Environment information

AKS - opentelemetry-operator

OpenTelemetry Collector configuration

Load Balancer collector config:

config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: localhost:4317
            include_metadata: true
            auth:
              authenticator: bearertokenauth

          http:
            endpoint: localhost:4318
            include_metadata: true
            auth:
              authenticator: bearertokenauth

    exporters:
      debug:
          verbosity: detailed
      loadbalancing:
        routing_key: "traceID"
        protocol:
          otlp:
            timeout: 1s
            auth:
              authenticator: headers_setter
            sending_queue:
              enabled: false

      headers_setter:
        headers:
          - action: upsert
            key: X-Scope-OrgID
            from_context: tenant_id
            default_value: default-tenant

    processors:
      batch:
        send_batch_size: 8388608
        send_batch_max_size: 16777216
        metadata_keys:
          - tenant_id
        metadata_cardinality_limit: 30

    exporters:
      debug:
          verbosity: detailed
      loadbalancing:
        routing_key: "traceID"
        protocol:
          otlp:
            timeout: 1s
            auth:
              authenticator: headers_setter
            sending_queue:
              enabled: false
            tls:
              insecure: true
              ca_file: /tls/ca.crt
              cert_file: /tls/tls.crt
              key_file: /tls/tls.key
              server_name_override: grafana-tempo.observability.svc.cluster.local
        resolver:
          dns:
            hostname: otel-external-collector-headless.observability.svc.cluster.local
            port: "4317"


Backend collectors config:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            include_metadata: true
            max_recv_msg_size_mib: 16777216
            tls:
              ca_file: /tls/ca.crt
              cert_file: /tls/tls.crt
              key_file: /tls/tls.key
          http:
            endpoint: 0.0.0.0:4318
            include_metadata: true

            tls:
              ca_file: /tls/ca.crt
              cert_file: /tls/tls.crt
              key_file: /tls/tls.key

    extensions:
      headers_setter:
        headers:
          - action: upsert
            key: X-Scope-OrgID
            from_context: tenant_id
            default_value: default-tenant

    processors:
      batch:
        send_batch_size: 8388608
        send_batch_max_size: 16777216
        metadata_keys:
          - tenant_id
        metadata_cardinality_limit: 30

    exporters:
      otlphttp/tempo:
        endpoint: "https://grafana-tempo-distributor.observability.svc.cluster.local:4318"
        auth:
          authenticator: headers_setter
        tls:
          insecure: false
          ca_file: /tls/ca.crt
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          server_name_override: grafana-tempo.observability.svc.cluster.local

    service:
      extensions:
       - headers_setter
      pipelines:
        traces:
          receivers:
           - otlp
          processors:
           - resourcedetection
           - batch
          exporters:
            - otlphttp/tempo
            - debug

Log output

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[meSATYA]

Seen same issue as well.

[atoulme]

@jpkrohling please review as codeowner