Applying memory_limiter extension

[dmitryax]

Problem

As part of #8632, we want receivers to be able to reject incoming requests if the collector is running out of memory limits instead of accepting the data transforming to OTLP and get it rejected by the memory_limiter processor. Scraping receivers would skip the scraping in that situation.

We introduced a memory_limiter extension in #8964, but it has not been applied yet.

#9397 proposes a way to configure a memory_limiter extension explicitly by receivers using httpconfig. Applying only this approach to other receivers would complicate the user configuration interface as they would have to explicitly connect every receiver with a memory_limiter extension.

Proposed solution

We can introduce an option to configure the memory_limiter extension in a way that it's applied to every receiver. Particular receivers can override the default memory limiter and connect to a different one.

Configuration example:

extensions:
  memory_limiter:
    limit_mib: 800
    apply_to_all_receivers: true
  memory_limiter/internal_metrics:
    limit_mib: 1000

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  prometheus:
    memory_limiter: memory_limiter/internal_metrics
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ["localhost:8888"]

processors:
  batch:

exporters:
  debug:
    verbosity: detailed

service:
  extensions: [memory_limiter, memory_limiter/internal_metrics]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [debug]

In this configuration, the default memory_limiter extension is implicitly applied to otlp receiver, and the additional memory_limiter/internal_metrics extension is applied explicitly to the prometheus receiver.

The user cannot define more than one memory_limiter extension with apply_to_all_receivers: true; the config validation will fail.

If memory_limiter extension with apply_to_all_receivers: true is defined, all the receivers used in the pipelines must support the memory_limiter extension. Otherwise, the collector won't start.

Proof of concept: #9590

[TylerHelmuth]

@dmitryax should this functionality exist in the receiverhelper, similar to how batching is being added to the exporterhelper?

[dmitryax]

@TylerHelmuth, yes, we can introduce a receiver helper. It doesn't exist at this point. It may be a cleaner implementation for the memory_limiter. If we think it can help with other problems in the future, we can add it. Otherwise, it can be also an unnecessary user interface complication. I'll look into that

[dmitryax]

First, I would like us to agree on the user interface and the way how the memory_extension can be enabled for all receivers

[TylerHelmuth]

Instead of needing apply_to_all_receivers: true, could the extension component named memory_limiter be applied to all unless the component specifies a different ID?

Pros:

• more backwards compatible
• users wouldnt need to remember to set the boolean to get the feature
• Less config

Cons:

• config is less self-describing

[dmitryax]

more backwards compatible

How is it more backward compatible? The extension is no-op at this point

How do you know which one is applied to all receives if I have let's say two memory_limiter extensions defined? Looking at each receiver at filtering out one that is not referenced by any of them? This seems like too much magic.

[jpkrohling]

I have a pending proposal for supporting interceptors (#7441) but I've failed to dedicate engineering time for a PoC. While I'm not too tied to that specific proposal, I think that this here could be split into two parts: one more generic being part of the Collector first, and one with the memory limiter being one implementation.

[TylerHelmuth]

How is it more backward compatible? The extension is no-op at this point

It is closer to what users do now with the processor, so changing it to an extension would be a little easier, but not significant.

This seems like too much magic.

Ya probably

[dmitryax]

one more generic being part of the Collector first

@jpkrohling what do you mean by this?

The middleware proposal makes sense to me, but it's not applicable to all receivers, e.g., scraping receivers. The memory_limiter is supposed to be supported by all of them. It's also important to apply the memory limiter before any other middleware for push-based receivers. So I think it's ok to keep them separate.

[jpkrohling]

one more generic being part of the Collector first

Sorry, I mean a general mechanism (interceptors/middleware/...), plus an implementation of it. I agree about the scraping one, though.

but it's not applicable to all receivers, e.g., scraping receivers

Stupid question, but isn't a memory limiter being used in a scraping receiver similar in usage to an exporter, in the sense that they both could intercept an outgoing HTTP/gRPC request? Perhaps there could be two interceptor interfaces, one for servers (typical receivers) and one for clients (scraping receivers and typical exporters)?

[dmitryax]

Stupid question, but isn't a memory limiter being used in a scraping receiver similar in usage to an exporter, in the sense that they both could intercept an outgoing HTTP/gRPC request? Perhaps there could be two interceptor interfaces, one for servers (typical receivers) and one for clients (scraping receivers and typical exporters)?

Scraping doesn't always involve making HTTP/gRPC requests to an external endpoint. They can read logs or metrics from local files. They can also use some 3rd-party libraries talking to external API, so it'll be hard to apply interceptors.

I think scraping receivers need to skip scrapes on the application level once the memory limit is reached and avoid even making attempts for any external calls.

[moh-osman3]

Hey @dmitryax @timannguyen thanks for all your work to create this memorylimiterextension and to make it useable by receivers. I just wanted to check in and see if there's anything I can do to help make this memorylimiterextension useable?

Our team is struggling with our service reliability because of memory/latency issues due to large requests being accepted by the receiver (despite using in flight bytes memory limiter we are applying in a custom processor). We want to use the core collector memorylimiterextension or create a custom extension to apply our limiter at the start of our pipeline. We use an otelarrow receiver which sets up a GRPC server and seems that we could make use of configgrpc and interceptors to apply the memorylimiterextension.

1. Is there anything blocking PR from being merged? [config/confighttp] add memorylimiterextension to confighttp #9397
2. May I implement something similar for configgrpc so our gRPC servers can leverage memory_limiter extensions?

[dmitryax]

Hi @moh-osman3, thank you for the interest. I'll be happy to work on this together.

#9673 and #9397 add a configuration option right in the httpconfig and grpcconfig. It's a good option if users want to enable memory_limiter explicitly. However, I believe it's important to provide a way to apply a memory_limiter extension to all the receivers used in the collector pipelines, as proposed in this issue.

I tried to prototype this approach in #9590, and it seems like it doesn't work with having memory_limiter option right in httpconfig and grpcconfig configs. Likely, every receiver has to have this option added separately in their config. That's why I'm hesitant to merge these PRs to avoid breaking them down the road. If you can find a way to merry httpconfig and grpcconfig config with the approach proposed in this PR, I'll be happy to move it forward.