Skip to content

Fix duplicate exported_job labels with transform processor #13239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

samuelarogbonlo
Copy link

@samuelarogbonlo samuelarogbonlo commented Jun 19, 2025

Description

This PR fixes an issue where Prometheus metrics (e.g., target_info) contain exported_job labels with namespace prefixes (e.g., production/my-app). This label change began after upgrading to OpenTelemetry Operator v0.125.0 and Collector v0.126.0, causing duplicate time series in Prometheus.

The solution introduces a transform processor that strips namespace prefixes from exported_job labels, ensuring consistent metrics and compatibility with existing dashboards and alerts.

It provides:

  1. A Dockerfile for building a custom OpenTelemetry Collector image
  2. A collector-config.yaml with the transform processor included
  3. A kubernetes-deployment.yaml example manifest for maintainers

Link to tracking issue

Fixes #13079

Testing

  1. Local testing with custom-built collector:
  • Confirmed that metrics previously labeled as production/my-app are now my-app
  • Verified Prometheus target and label consistency:
    curl http://localhost:8889/metrics
    Confirmed no duplicate exported_job label values.
  1. Health check verified:
    curl http://localhost:13133
    {"status":"Server available","upSince":"...","uptime":"..."}
    Image builds and runs cleanly:
  • Custom collector compiled and runs with no config errors
  • No unintended metrics dropped

Documentation

This PR includes:

  • Dockerfile: Custom OpenTelemetry Collector image with the transform processor
  • collector-config.yaml: Configuration demonstrating the namespace-stripping fix
  • kubernetes-deployment.yaml: Example deployment manifest for maintainers

Detailed deployment and testing instructions are in this PR description for maintainers to implement or adapt as needed.

Notes

This solution provides maximum flexibility:

  • Choose the deployment option that best fits infrastructure
  • No changes required to existing Prometheus or dashboard configurations
  • Can be deployed immediately or integrated into existing collector configs
  • Comprehensive testing instructions included

The transform processor approach is the least disruptive solution that maintains full backward compatibility while solving the duplicate time series issue.

- Add transform processor to strip namespace prefixes from exported_job labels
- Resolves open-telemetry#13079: duplicate time series after OpenTelemetry v0.125.0/v0.126.0 upgrade
- Provides 3 deployment options for maintainers (inline config, custom image, k8s manifest)
- Maintains backward compatibility with existing dashboards and alerts
- Includes comprehensive testing and implementation documentation

Files added:
- Dockerfile: Custom collector with transform processor
- collector-config.yaml: Complete configuration with namespace-stripping fix
- kubernetes-deployment.yaml: Ready-to-deploy Kubernetes manifest
@samuelarogbonlo samuelarogbonlo requested a review from a team as a code owner June 19, 2025 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Exported_job label in metrics (e.g., target_info) started including the namespace as part of the value
1 participant