Endorse use of runtime-defined instrumentation scope attributes

[jade-guiton-dd]

What are you trying to achieve?

The Collector SIG is planning to add attributes to the Collector's internal telemetry to identify which component instance within the Collector emitted the telemetry. For practical reasons, instrumentation scopes were thought to be the most appropriate attribute collection to include these "component attributes" in. (For more context, see this Collector issue and this Go SIG issue.)

As discussed in the Spec SIG meeting on 2025-03-11, instrumentation scopes are generally used to separate telemetry at the source code level (ie. which library emitted it, and which version thereof). On the other hand, our component attributes are defined at runtime, based on the Collector's configuration, and may be applied to telemetry from multiple instrumentation libraries. However, because they stay static after the instantiation of the Collector pipeline, the general consensus seemed to be that our use case would fall within the intended semantics of scope attributes.

However, there are still a number of places in the specification that imply a narrow, source-level definition of instrumentation scopes:

• The Glossary defines a scope as "a logical unit of the application code", and states that "since the scope is a build-time concept the attributes of the scope cannot change at runtime."
• The Log, Trace, and Metric API definitions all link to the glossary entry, and mention "the instrumentation scope, such as the instrumentation library, package, module or class."

(The docs page about scopes under "Concepts" uses very similar language, but that one is not in the specification proper.)

It's possible that nothing strictly normative prevents the use of runtime-defined scopes or scope attributes, but we believe it would be good to loosen the language regarding instrumentation scopes, and perhaps even add an example to endorse use cases similar to ours.

[carlosalberto]

So I did a little digging and found this:

• Glossary is non-normative documentation piece, as it DOES NOT contain normative language itself (probably worth moving some definitions out of it, and into the Spec?).
  • Hence the "since the scope is a build-time" can be dropped/relaxed?
• Some SIGs do support InstrumentationScope as a de facto runtime concept, i.e. you won't get any cached Tracer with the attributes specified in the first call, but a Tracer with the attributes passed at each invocation:
  • Python: https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py#L1262
  • Go: https://github.com/open-telemetry/opentelemetry-go/blob/main/sdk/trace/provider.go#L145
• The Prometheus compatibility layer relies on InstrumentationScope's name and version be a single instance, i.e. no more than one element with different attributes.
  • However, this was 'broken' since Update Instrumentation Scope glossary entry with correct identifying fields #4244 included attributes as identifying for InstrumentationScope.

IMHO other than the Prometheus part (which needs to be adjusted/updated anyway), we should be be good to make InstrumenationScope a runtime concept, rather than a compile-time one. Would love to hear the opinion from maintainers.

[jmacd]

As for the Prometheus compatibility issue, I feel that there is a solution based on EntityRefs. If the scope definition contains an entity definition, then the identifying fields of the scope entity will be declared in a way that Prometheus compatibility can be maintained.

To Prometheus users, the otel_scope_info metric can still be used, but the PromQL join operation (e.g., group_left) will have to specify each of the identifying scope attributes in order to effectively join the data, and each of the identifying scope attributes will be copied into the metric attributes.

[jmacd]

Since the problem mentioned above ("other than the Prometheus part") is already a real problem (open-telemetry/opentelemetry-collector#12917), we should plot a path forward.

@carlosalberto @jsuereth

Concretely, there is a question here about whether we can use entities to describe scopes. As soon as we allow instrumentation scopes to define multiple attribute sets, we also need a way to create multiple otel_scope_info metrics. Moreover, each metric instrument needs to add attributes that correlate with the correct scope info metric; this is done using the otel_scope_name and otel_scope_version attributes. Here, for the OTel-Go exporter, https://github.com/open-telemetry/opentelemetry-go/blob/2e49a0d34adca0c4c0c9e80cba354ad1e060c1bf/exporters/prometheus/exporter.go#L217, we will need to add attributes in the metric that can be joined with the correct otel_scope_info metric.

[pellared]

@jmacd, potentially related issue:

• [Prometheus and OpenMetrics] Handle scheme URL and scope attributes as identifying #4223

Please "hide" or even "remove" my comment if it is not helpful as I was not going through this issue in details.