Description
Component(s)
exporter/kafka
Is your feature request related to a problem? Please describe.
The current Kafka exporter does not produce any internal operational metrics, which makes it difficult to operate and monitor. Specifically, without metrics that measure the fundamentals such as throughput, error rate, and latency, it is challenging to use the exporter reliably.
Describe the solution you'd like
I'd like to propose to add internal metrics to the Kafka exporter through metadata.yaml and have code for them generated. Then instrument both currently supported clients Sarama and franz-go, to report these metrics.
Describe alternatives you've considered
No response
Additional context
I'd like to propose to add the following list of the metrics.
Click to expand
| Name | Description | Type |
|---|---|---|
| kafka_exporter_records | The number of exported records | sum |
| kafka_exporter_bytes | The number of exported bytes, accounting for client compression | sum |
| kafka_exporter_bytes_uncompressed | The number of exported uncompressed bytes | sum |
| kafka_exporter_latency | Time it took to export a batch of records | histogram |
| kafka_broker_connects | The number of connections opened | sum |
| kafka_broker_disconnects | The number of connections closed | sum |
| kafka_broker_failed_connects | The number of connection errors | sum |
| kafka_exporter_throttling_duration | The throttling duration imposed by the broker when exporting messages | histogram |
Note that franz-go client allows for a greater level of introspection due to its advanced hook mechanism, and potentially even kotel plugin provided by the library could be used to add the metrics described above. For Sarama however only some of the metrics will be available.