Filter Processor should support inferred context statements

[jaronoff97]

Component(s)

pkg/ottl, processor/filter

Is your feature request related to a problem? Please describe.

With the introduction of passed in context to the filterottl package and inferred context for statements it would be great to bring these together for inferred context for conditions so that the filterprocessor (and the routing processor/connector eventually) could take advantage of the new flat-statement syntax.

Describe the solution you'd like

Ideally the configuration for the filterprocessor could be modified similar to the transform processor like so:

processors:
  filter/ottl:
    error_mode: ignore
    metrics:
      metric:
          - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"'
          - 'type == METRIC_DATA_TYPE_HISTOGRAM'
      datapoint:
          - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
          - 'resource.attributes["service.name"] == "my_service_name"'

to

processors:
  filter/ottl:
    error_mode: ignore
    metric_conditions:
          - 'metric.name == "my.metric" and resource.attributes["my_label"] == "abc123"'
          - 'metric.type == METRIC_DATA_TYPE_HISTOGRAM'
          - 'datapoint.attributes["hello"] == "world"'
          - 'resource.attributes["service.name"] == "my_service_name"'

This would allow a user to set both methods simultaneously keeping the functionality opt-in by default with a syntax matching that of the transform processor. If there are other recommendations for configuration, do let me know.

Describe alternatives you've considered

No response

Additional context

I'm planning on working on this.

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley @edmocosta
• processor/filter: @TylerHelmuth @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jaronoff97]

Going to jot down some implementation thoughts here:

1. Making the ParserCollection more generic to take in []string

Ultimately, i don't think there's much of a difference between the parsing of statements and contexts, both are potentially valid OTTL statements represented as a slice of strings. Currently the StatementsGetter interface is used which has a single method for getting the []string statements. I was thinking that we could remove the StatementsGetter interface (all of this code is marked subject for change!) and change the signatures to just take in a string slice. This would mean we don't have to add any new code, it's mostly just renaming internally and then modify the callers. If we didn't want to remove the interface and instead keep what's there for now, deprecating the existing methods and having them call the new ones

2. Add methods for ParseConditions and ParseConditionsWithContext

This would basically just be forking the methods that exist today and maybe abstracting the logic there today to work for both. This would also necessitate the creation of a new ConditionsGetter interface. After that, I think the rest is straightforward duplications

3. Modify the StatementsGetter interface to include conditions

Instead of making a solo interface for parsing conditions and a separate one for parsing statements, we could modify the existing interface to have methods for GetConditions() and GetStatements() given that the transformprocessor (among other processors) has a use case for both. The ParserCollection would then skip parsing for when there are no conditions or when there are no statements respectively (though it would error if they're both empty). This approach feels too cumbersome IMO, it couples the logic unnecessarily and would result in a potentially clunkier dev experience.

4. Make inferring context public

All of the above would be modifying the existing parser collection, but there already exists logic in the filterottl to create an Expr based on conditions, the issue is that those methods require known context. If we exposed the infer logic used by the parser collection to simply get the inferred context we could just pass that to the filterottl package for use there. This is probably the simplest way of achieving the goal, and would probably the most minimal set of changes. ... In fact, this is probably the simpler version of 1.

Collected thoughts

A key difference between the filterprocessor and the transformprocessor is that the latter is structured in terms of creating "consumers" for each context which are responsible for handling their own changes via ConsumeXXXXX(ctx context.Context, tX pX, cache *pcommon.Map). The filter processor, by contrast, is much simpler – you create a filterXXXXXProcessor instance which has its own processMetrics(ctx context.Context, tX pX) and works through the RemoveIf methods. As a result, method 4 is probably the fastest/easiest way to achieve the goal here, but I'm not positive it's the right approach. The v

[TylerHelmuth]

Making the ParserCollection more generic to take in []string

I support using []string instead of StatementsGetter.

Modify the StatementsGetter interface to include conditions

I prefer keeping separate structs for statements and conditions, like we do today with StatementSequence and ConditionSequence

Make inferring context public

Make conditions infer context will likely need changes in OTTL and then changes in ottlfilter. Any component wanting to do "filtering" with OTTL will continue to use ottlfilter.

[edmocosta]

The main idea behind the StatementsGetter interface is to allow API users to pass their own objects as a parameter, including custom objects such as the common.ContextStatements used by the transform processor. Those custom objects are useful for the ottl.ParsedStatementConverter callbacks, that can leverage the custom object (StatementsGetter) to receive extra arguments required to transform K into R (see transformprocessor/internal/common/logs.go#L95 for an example).

That said, I think the StatementsGetter interface is not an issue for supporting conditions, what I believe we need to think is how we can shape the parser collection to also supports conditions, and that essentially means how can we transform parsed ottl.Condition[K] into something common R? Should it support some kind of converter function like it does for statements? perhaps adding an extra ParsedConditionConverter argument in the ottl.WithParserCollectionContext, or maybe making them Options so it does not require users to provide a value if the parser collection is only meant to parse conditions?

Besides that, I think we've everything needed in place already.

[TylerHelmuth]

I forgot about common.ContextStatements implementing the interface.

@edmocosta It would be nice to add some Condition capabilities to the existing ParserCollection. That would mimic closely how the ottl.Parser has a ParseStatements and ParseConditions.

[edmocosta]

Yes, exactly! the options I mentioned before would look more or less like this:

pc, err := NewParserCollection[any](settings,
    WithParserCollectionContext("foo", fooParser, ottl.WithParserCollectionStatementsConverter(convertFooStatements), ottl.WithParserCollectionConditionsConverter(convertFooConditions)),
)

So when it invokes pc.ParseXXX it would require the specific converter for the method being executed, if that's not present, we could just return an error. The good thing about this approach is not requiring users to provide both converters when they don't need to.

Another option would be just adding an extra "conditions converter" argument to the WithParserCollectionContext option, something like:

pc, err := NewParserCollection[any](settings,
    WithParserCollectionContext("foo", fooParser, convertFooStatements, convertFooConditions),
)

Finally, for both options, a few extra parser methods to prepend the context to conditions paths will be needed, but that shouldn't be complicated.