How to optimize Kafka receiver for slow consumption of messages?

[addname]

Component(s)

receiver/kafka

Describe the issue you're reporting

Kafka receiver is only consuming a few dozen messages per second, how can I optimize it?

receivers:
  kafka/1:
    brokers:
    - 10.xx.xx.x1:9092
    - 10.xx.xx.x2:9092
    - 10.xx.xx.x3:9092
    protocol_version: 2.0.0
    topic: otlp-spans-topic-01
    group_id: otel-collector-01
    encoding: jaeger_json      
  kafka/2:
    brokers:
     - 10.xx.xx.x1:9092
     - 10.xx.xx.x2:9092
     - 10.xx.xx.x3:9092
    protocol_version: 2.0.0
    topic: otlp-spans-topic-02
    group_id: otel-collector-02
    encoding: jaeger_json  

service:
    traces:
      exporters:
      - clickhousetraces
      processors:
      - signozspanmetrics/prometheus
      - batch
      receivers:
      - otlp
      - kafka/1
      - kafka/2
      - jaeger 
 

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[shadowofmoo]

Does kafka have tls enabled?

[crobert-1]

/label waiting-for-author

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[gotosre]

+1, now only can only add partitions to raise consume rate

[crobert-1]

@gotosre Can you share the full configuration you're using, and explain the problem you're running into?