Skip to content

Tail-sampling evaluation completing but not filtering out any traces for Grafana Tempo Distributed  #27049

Closed
@gmcwhirt

Description

@gmcwhirt

Component(s)

No response

What happened?

I am not entirely sure if this a bug or not, but I am using tail sampling to filter out basic endpoints for Grafana Tempo Distributed.

Steps to reproduce
I don't have the exact steps to reproduce this config not to work, but right now my log is saying this:

level=debug component=otelcol.processor.tail_sampling.tempoFilter msg="Sampling policy evaluation completed" batch.len=0 sampled=0 notSampled=0 droppedPriorToEvaluation=0 policyEvaluationErrors=0

My understanding is that since the batch.len is = 0, that for whatever reason my traces are not getting to my batch. I am using the receiver and batch Otel.processor as well and they are set up for the receiver to send the output to the batch and has batch sending enabled..

What did you expect to see?
I expect for my logs to show that it is filtering out some of these traces that match the endpoints in the config. I have tried invert match and that did not work either...

What did you see instead?
A clear and concise description of what you saw instead.

What version did you use?
Kubernetes: 2.21.1

Here is the config that I am using:

otelcol.processor.tail_sampling "tempoFilter" {
policy {
name = "generic-endpoint-filtering"
type = "string_attribute"

      string_attribute {
      key                    = "http.url"
      values                 = ["/health", "/info","/prometheus" ]
      enabled_regex_matching = true
      invert_match           = true
      }
    }
    buffer_duration = 60 seconds
    sampling {
       type = "head"
    }

    policy {
      name = "generic-method-filtering"
      type = "string_attribute"

      string_attribute {
      key                    = "http.method"
      values                 = ["OPTIONS"]
      enabled_regex_matching = true
      invert_match           = true
      }
    }

Environment
MAC Ventura 13.4

Additional context
Add any other context about the problem here.

Collector version

Kubernetes: 2.21.1

Environment information

Environment
MAC Ventura 13.4

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions