otlp http exporter doesn't handle 429s properly

[jaronoff97]

Describe the bug
Currently, when the OTLP HTTP exporter's request is responded to with any HTTP error code (429 for example), the status code is dropped which makes back propagation impossible. This is because we create a new formatted error and return that to the exporter's caller.

Steps to reproduce
Setup a collector with any receiver, no processors, and an OTLP http exporter. Have the exporter send to a server that always responds with a 429. Send data to the receiver, the status code will be a 500.

What did you expect to see?
I expected the receiver to observe a 429.

What did you see instead?
The receiver observed a 500.

What version did you use?
v0.97.0

What config did you use?

exporters:
  otlphttp:
    logs_endpoint: "destination"
    sending_queue:
      enabled: false
    retry_on_failure:
      max_elapsed_time: 20s
  logging:
    verbosity: detailed
receivers:
  webhookevent:
    endpoint: "0.0.0.0:8080"
    include_metadata: true
processors:
service:
  pipelines:
    logs:
      exporters:
      - otlphttp
      processors: []
      receivers:
      - webhookevent
  telemetry:
    metrics:
      level: detailed
    resource:
      service.name: test

Environment
"Ubuntu 20.04"

Additional context

[TylerHelmuth]

I believe the correct behavior is for the otlphttpexporter to return permanent or non-permanent error based on the response from the client.

The body of the response should include a grpc Status object, this could be used as part of the returned error.

[TylerHelmuth]

@jaronoff97 looking at the exporter again, it will only return a permanent error (specifically a 500) if the response code from the server is not retryable. If it is retryable it returns a throttleRetry error and the retry sender will try again. Eventually the retry sender will give up and return a basic error that will get treated by the otlpreceiver as a 500.

So there are 2 situations:

1. Retry mode is enabled
   • I believe this situation is handled correctly in the sense that the exporter should retry. After the max number of retries is hit it isn't clear to me what should happen. Right now we return a basic error and the otlpreceiver would treat that like a 500.
2. Retry mode is disabled
   • I believe this situation is handled incorrectly since the server response of 429 is not correctly passed back up the collector pipeline to the otlpreceiver.

[TylerHelmuth]

@dmitryax @evan-bradley what are your thoughts?

[jaronoff97]

that makes sense – in this case the collector is p much a proxy to its caller so we never want to retry and always propagate that back to the user

[jaronoff97]

Right now that's not even respected too – we just assume that we want to retry any 429 which then will always cause a 500