[exporter/loadbalancing] Add the ability to load balance by Metric stream ID

[RichieSams]

Component(s)

exporter/loadbalancing

Is your feature request related to a problem? Please describe.

The LoadBalancing exporter is a great idea. It currently exposes a number of ways to route metrics coming in:

const (
	traceIDRouting routingKey = iota
	svcRouting
	metricNameRouting
	resourceRouting
)

However, there are many situation where these routing methods can lead to non-uniform distributions of metrics. For example, if datapoints use lots of labels, or med to high cardinality labels. The finest grain routing we can currently do is metric name routing. But if the majority of the datapoints are all the same metric name, then this doesn't help.

Describe the solution you'd like

I would like to propose adding a new value to the routing enum: streamIDRouting. This would route individual datapoints based on their unique ID. We could use the new internal/exp/metrics/identity package: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/internal/exp/metrics/identity/stream.go#L29

I'm willing to be the implementer

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

@RichieSams Hi, Thanks for the proposal. It's yours

[skandragon]

I am also needing something similar to this, but likely need to dig a bit deeper and make this per-metric-attribute.

My use case is, I have a custom processor that decorates telemetry with an attribute, placed either on the log, metric, or span. This key is what I want to partition on, and send to each back-end to be gathered and ultimately written to the same s3 output file.

What is the recommended way to add this, as the PR associated with this change seems to be rejected? I know the PR is not sufficient as it only looks at resource attributes, and I would need to look deeper, and I assume split deeper as well.

[jpkrohling]

I believe @RichieSams will work on this, the PR was closed in favor of breaking it into smaller chunks of work.

[RichieSams]

Correct. I already split up the commits, I just got busy at work last week. I'll create new PRs today

[RichieSams]

This is the first PR. Just a trivial helper function: #32794

Then there are these 4 commits that will be applied on top. @jpkrohling How do you want me to do them? I can serially make each a PR, get it merged, then go on to the next. Or do you want to combine any?

1. chore: Add str const variables for the routingKeys RichieSams@bb473e1
2. [exporter/loadbalacing] Refactor how metrics are split and then re-joined after load-balacing RichieSams@89a14af
3. [exporter/loadbalancer] Refactor the metrics export benchmarks RichieSams@891bdfa
4. [exporter/loadbalancer] Add a new routing key: streamID RichieSams@5314b93