New component: Alert Manager Exporter

[sokoide]

The purpose and use-cases of the new component

OTEL event is a span event in Tracing added by AddEvent API or a logging event added by Log Event API.
Before the OTEL logging is finalized, we are using the former (trace event) for events.

We'd like to filter and forward specific OTEL events to an Alert Manager instances to notify a critical error which need attention.

The filter part can be done by the existing transform processor. This component will focus on the exporter part to Alert Manager.

Example configuration for the component

exporters:
  logging:
    loglevel: debug
  alertmanager:
    generator_url: "otelcol.example.com:4317"
    endpoint: "http://alertmanager.example.com:12345"
    severity: "info"
    severity_attribute: "severity"
    tls:
      insecure: true
    timeout: 1s
    sending_queue:
      enabled: true
      num_consumers: 2
      queue_size: 10
    retry_on_failure:
      enabled: true
      initial_interval: 10s
      max_interval: 10s
      max_elapsed_time: 10s

Telemetry data types supported

• traces (initial focus)
• logs (later)

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

No response

Additional context

No response

[jpkrohling]

cc @nicolastakashi, who proposed an Alert Manager receiver and exporter in the past (#18526)

[jpkrohling]

@sokoide, there was an earlier proposal for this, and the consensus back then seems to be that an exporter would have to suite one of the three signals we have today (logs, metrics, traces). Alerting wouldn't fit cleanly into any one of those, and using logs for that would not be acceptable.

That said, please take a look at that earlier proposal and let us know if there's anything different about this proposal here that would possibly change our view of things.

[sokoide]

@jpkrohling , thank you for taking a look and the explanation. It looks to me that #18526 is a generic alert manager receiver/exporter discussion.
Our request is more specific to event alerting which a part of OTEL tracing now, may be a part of OTEL logging in the future. We have an enterprise scenario to send OTEL events and alert from some of them when it meets alerting criteria.

This is not a proposal of adding an alert as a data type, but more of flow-control of a specific type of trace (OTEL event).
For example, our traced applications call OTEL AddEvent API to send unexpected exceptions (e.g. failed to query a DB even after 3 retries in a business application). The API is great to record the occurrences. However, there is no proactive way to notify us for critical trace events. We'd like to alert from it which is the motivation of the component request.
Would that meet criteria of a OTEL component?

[mcube8]

Diagram for the exporter flow -

The exporter would capture SpanEvents from otlp Trace and convert it to Alert Payloads for Alertmanager.

[nicolastakashi]

I see having a better fit for this #20888
But the doesn't seem to me we will have this soon, and an alternative of that is to use a https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md and send it to your own service.

Although I would also prefer to have this as part of the pipeline in a more native way

[sokoide]

@nicolastakashi, thank you very much for your explanation and the suggestions. We understood that the alert exporter doesn't meet the otelcol component criteria. We'll move the alert exporting logic from otelcol (receiver) into another gRPC server and connect them with the OTLP gRPC exporter. When #20888 is implemented, we'll move onto it.
Thank you very much!

[jpkrohling]

We discussed this proposal today during the SIG Collector, and there were no concerns by the maintainers. They asked @gouthamve to express his opinions here as a Prometheus WG, but other than that, there's a green light for this to move forward.

I'm sponsoring this component.