Description
Component(s)
exporter/prometheusremotewrite
Is your feature request related to a problem? Please describe.
Prometheus remote write exporter does not currently retry on HTTP 429s TooManyRequests. Many remote write destinations implement 429 and it's not great if we're dropping metrics because we're being temporarily rate limited.
For context, the remotewrite spec says:
Prometheus Remote Write compatible senders MUST retry write requests on HTTP 5xx responses and MUST use a backoff algorithm to prevent overwhelming the server. They MUST NOT retry write requests on HTTP 2xx and 4xx responses other than 429. They MAY retry on HTTP 429 responses, which could result in senders "falling behind" if the server cannot keep up. This is done to ensure data is not lost when there are server side errors, and progress is made when there are client side errors.
Describe the solution you'd like
The prometheus reference implementation here retries on 429:
https://github.com/prometheus/prometheus/blob/main/storage/remote/client.go#L239-L242
It also uses the 'retry after duration' header to modulate the retry backoff -- not sure how best to handle that in the otel implementation.
The current implementation retries on 500, so I figure we could just add a 429 check to this same if statement, at minimum.
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/exporter.go#L270-L273
Describe alternatives you've considered
No response
Additional context
No response