Prometheus receiver fails on federate endpoint when job and instance labels are missing

[pavolloffay]

Component(s)

receiver/prometheus

What happened?

Description

The Prometheus receiver fails to scrape the federate endpoint when honor_labels: true and the target metric does not have instance and job labels, which can be the case for aggregated metrics.

Log message:

2024-04-19T09:26:29.254Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1713518789233, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}

Steps to Reproduce

    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'federate'
              scrape_interval: 5s
              scheme: https
              tls_config:
                ca_file: /etc/pki/ca-trust/source/service-ca/service-ca.crt
              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              honor_labels: true
              params:
                'match[]':
                  - '{__name__=~"cluster:usage:containers:sum"}'

Expected Result

Working scraping of the federate endpoint with honor_labels: true for metrics that don't have instance and job labels.

Actual Result

2024-04-19T09:26:29.254Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1713518789233, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}

From

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/transaction.go

Line 129 in 13fca79

t.logger.Warn("Failed to scrape Prometheus endpoint",

Collector version

0.93.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
Kubernetes

OpenTelemetry Collector configuration

prometheus:
        config:
          scrape_configs:
            - job_name: 'federate'
              scrape_interval: 5s
              honor_labels: true
              params:
                'match[]':
                  - '{__name__=~"cluster:usage:containers:sum"}'
                  - '{__name__=~"cluster:cpu_usage_cores:sum|cluster:memory_usage_bytes:sum"}'
                  - '{__name__=~"workload:cpu_usage_cores:sum|workload:memory_usage_bytes:sum"}'
                  - '{__name__="namespace_memory:kube_pod_container_resource_limits:sum|namespace_memory:kube_pod_container_resource_requests:sum|namespace_cpu:kube_pod_container_resource_limits:sum|namespace_cpu:kube_pod_container_resource_requests:sum|"}'
              metrics_path: '/federate'
              static_configs:
                - targets: 
                  - "prometheus-k8s.openshift-monitoring.svc.cluster.local:9091"

Log output

2024-04-19T09:26:29.254Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1713518789233, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}

Additional context

• Prometheus Receiver - honor_labels set to true doesnt work with federation #5757
• federate docs suggest setting honor_labels: true https://prometheus.io/docs/prometheus/latest/federation/

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Shouldn't a federated endpoint always have job + instance labels?

[pavolloffay]

I am not sure, it is not the case for aggregated in-clsuter metrics on OpenShift. Is the requirement documented somewhere?

[dashpole]

How do you know that honor_labels is the cause? The error message indicates that the receiver failed to scrape the endpoint, and looks like it already has a job + instance label. I believe you need to enable debug logging to see the detailed error message, since it is in the prometheus server code at debug level.

[pavolloffay]

After switching honor_labels: false the log message from the collector disappeared and the collector could scrape the metrics from the Prometheus federate endpoint.

[dashpole]

Interesting... Can you enable debug logs to see what the error is that causes the scrape to fail?

[pavolloffay]

There is only warning log that is pasted in the first comment.

[pavolloffay]

I was able to reproduce it again for scraping a different metric:

    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'federate'
              scrape_interval: 5s
              honor_labels: true
              params:
                'match[]':
                  - '{__name__="kube_pod_container_state_started"}'
              metrics_path: '/federate'
              static_configs:
                - targets: 
                  - "prometheus-k8s.openshift-monitoring.svc.cluster.local:9091"

The metric

kube_pod_container_state_started{container="alertmanager",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="alertmanager-main-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4787842f-88ad-4d42-9611-8939f3adde8c"} 1.715085997e+09
kube_pod_container_state_started{container="alertmanager-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="alertmanager-main-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4787842f-88ad-4d42-9611-8939f3adde8c"} 1.715085997e+09
kube_pod_container_state_started{container="approver",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-network-node-identity",pod="network-node-identity-qjmkf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a89de87d-7ac8-4372-bd63-9c6c4e1c7e23"} 1.715085497e+09
kube_pod_container_state_started{container="authentication-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-authentication-operator",pod="authentication-operator-596ffb9669-hbg2p",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="ddecf886-fd4d-4e45-9ea8-dcab1dfed67a"} 1.715085513e+09
kube_pod_container_state_started{container="catalog-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="catalog-operator-69f8747454-b54sn",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a487686d-7b9d-49ba-a7d2-8c93f8575302"} 1.715085505e+09
kube_pod_container_state_started{container="check-endpoints",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-network-diagnostics",pod="network-check-source-694f49c6d7-69x4l",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a3e5aa61-74d8-440c-8651-33b922e60956"} 1.715085515e+09
kube_pod_container_state_started{container="cluster-image-registry-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-image-registry",pod="cluster-image-registry-operator-74cbd85779-cnjsf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c8378206-10ba-4387-a985-6209ee54d7b8"} 1.715085506e+09
kube_pod_container_state_started{container="cluster-monitoring-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="cluster-monitoring-operator-6dd894bb4b-9jq84",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="bdf1a88e-5148-410d-8bea-a8d39ab1c38c"} 1.715085911e+09
kube_pod_container_state_started{container="cluster-policy-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-controller-manager",pod="kube-controller-manager-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f9fcfc9a-0ca2-482d-96dd-0b5dc62dd8f2"} 1.715085477e+09
kube_pod_container_state_started{container="cluster-samples-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-cluster-samples-operator",pod="cluster-samples-operator-f9ff458d-2dxmd",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2f5bc149-ab18-4126-b25c-c6f405b390f4"} 1.715085513e+09
kube_pod_container_state_started{container="cluster-samples-operator-watch",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-cluster-samples-operator",pod="cluster-samples-operator-f9ff458d-2dxmd",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2f5bc149-ab18-4126-b25c-c6f405b390f4"} 1.715085515e+09
kube_pod_container_state_started{container="cluster-version-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-cluster-version",pod="cluster-version-operator-6bbdb9fbd-c2g9p",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f7e53684-243f-4ba6-8346-aa8ff39cb5a7"} 1.715085496e+09
kube_pod_container_state_started{container="collect-profiles",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="collect-profiles-28584930-rdw4t",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="3f05910f-84a0-4460-9d5a-adfbf6902018"} 1.715095802e+09
kube_pod_container_state_started{container="collect-profiles",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="collect-profiles-28584945-mtfsj",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4145b528-d3a4-46e4-8d32-ef81a65b4bc6"} 1.715096702e+09
kube_pod_container_state_started{container="collect-profiles",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="collect-profiles-28584960-fmks9",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="5c585c4c-5e5a-419d-a0b6-187240477a60"} 1.715097604e+09
kube_pod_container_state_started{container="config-reloader",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="alertmanager-main-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4787842f-88ad-4d42-9611-8939f3adde8c"} 1.715085997e+09
kube_pod_container_state_started{container="config-reloader",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-k8s-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6892a658-35f7-4c6f-8735-fee9163205ae"} 1.715085994e+09
kube_pod_container_state_started{container="config-reloader",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81a65bf1-d96e-41a7-b624-c88b6e0e0767"} 1.715085993e+09
kube_pod_container_state_started{container="config-reloader",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="thanos-ruler-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="98403218-82a1-424a-8eb1-561ff3f96b0d"} 1.715085983e+09
kube_pod_container_state_started{container="console",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-console",pod="console-fc69c78fc-hts4d",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="40fb1469-c393-4ec0-a55e-22a3812e137a"} 1.715086015e+09
kube_pod_container_state_started{container="console-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-console-operator",pod="console-operator-756d76964d-5f9fr",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="024a8c12-8eb1-40db-a2db-d4f05f81a9a2"} 1.715085506e+09
kube_pod_container_state_started{container="control-plane-machine-set-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="control-plane-machine-set-operator-6bc95557dc-p5hm7",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a1e85f16-4560-4b79-9a5a-0345c2ecff68"} 1.715085512e+09
kube_pod_container_state_started{container="controller-manager",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-controller-manager",pod="controller-manager-7f5598846d-7vtq4",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9090c601-2e12-4424-b1ee-d4731a88ce03"} 1.715085561e+09
kube_pod_container_state_started{container="conversion-webhook-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-console-operator",pod="console-operator-756d76964d-5f9fr",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="024a8c12-8eb1-40db-a2db-d4f05f81a9a2"} 1.715085506e+09
kube_pod_container_state_started{container="csi-provisioner",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="hostpath-provisioner",pod="csi-hostpathplugin-vqcc7",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="e2fc6603-94ba-46e8-99c8-c2ad4eab7202"} 1.715085518e+09
kube_pod_container_state_started{container="dns",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-dns",pod="dns-default-cpdsn",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9a5f8f86-ae78-4b1b-91c1-ad298572ee95"} 1.715085505e+09
kube_pod_container_state_started{container="dns-node-resolver",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-dns",pod="node-resolver-z6hwv",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="04cf097f-bcbb-42e5-8a3f-185fe184cc7f"} 1.715085496e+09
kube_pod_container_state_started{container="dns-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-dns-operator",pod="dns-operator-5d8b885cc7-k9sxq",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="b27dc45e-8b72-4757-a36c-5d3cab233853"} 1.715085509e+09
kube_pod_container_state_started{container="download-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-console",pod="downloads-867ccb4fb9-mr9sj",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="474d4f14-f0f2-4a18-bd54-f62a6bfcf1b7"} 1.715085506e+09
kube_pod_container_state_started{container="etcd",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-etcd",pod="etcd-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="fe88f17b-961b-4429-8c66-168f8b04c97c"} 1.715085479e+09
kube_pod_container_state_started{container="etcd-metrics",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-etcd",pod="etcd-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="fe88f17b-961b-4429-8c66-168f8b04c97c"} 1.715085479e+09
kube_pod_container_state_started{container="etcd-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-etcd-operator",pod="etcd-operator-544455786b-rwsf5",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="1fca0a4b-56e5-4f71-9023-8da37b454016"} 1.715085511e+09
kube_pod_container_state_started{container="etcd-readyz",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-etcd",pod="etcd-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="fe88f17b-961b-4429-8c66-168f8b04c97c"} 1.715085479e+09
kube_pod_container_state_started{container="etcdctl",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-etcd",pod="etcd-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="fe88f17b-961b-4429-8c66-168f8b04c97c"} 1.715085479e+09
kube_pod_container_state_started{container="extract",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="08bd949d5e0a4686c4d9b0dc19ab6104947279aa2c43de27b9e24b020fqzpcw",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9f05223f-c5ee-4531-b6c2-6eaf82af6bc0"} 1.715092271e+09
kube_pod_container_state_started{container="extract",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="292e5c1db1d397e0d4b0573ef7ef9002a6ccd3dd471876d5e72907a6b45558s",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2f440282-3ea5-4ece-a64c-a796b06ffa20"} 1.715095454e+09
kube_pod_container_state_started{container="hostpath-provisioner",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="hostpath-provisioner",pod="csi-hostpathplugin-vqcc7",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="e2fc6603-94ba-46e8-99c8-c2ad4eab7202"} 1.715085512e+09
kube_pod_container_state_started{container="ingress-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-ingress-operator",pod="ingress-operator-8446b9df4c-5jm72",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="85236644-7617-4ab3-bf2d-6e0761091855"} 1.715085632e+09
kube_pod_container_state_started{container="kube-apiserver",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-apiserver",pod="kube-apiserver-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="127190b5-b7e6-487f-935d-865b89498a3e"} 1.715085477e+09
kube_pod_container_state_started{container="kube-apiserver-cert-regeneration-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-apiserver",pod="kube-apiserver-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="127190b5-b7e6-487f-935d-865b89498a3e"} 1.715085478e+09
kube_pod_container_state_started{container="kube-apiserver-cert-syncer",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-apiserver",pod="kube-apiserver-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="127190b5-b7e6-487f-935d-865b89498a3e"} 1.715085477e+09
kube_pod_container_state_started{container="kube-apiserver-check-endpoints",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-apiserver",pod="kube-apiserver-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="127190b5-b7e6-487f-935d-865b89498a3e"} 1.715085478e+09
kube_pod_container_state_started{container="kube-apiserver-insecure-readyz",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-apiserver",pod="kube-apiserver-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="127190b5-b7e6-487f-935d-865b89498a3e"} 1.715085478e+09
kube_pod_container_state_started{container="kube-apiserver-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-apiserver-operator",pod="kube-apiserver-operator-6968cf8cb5-fbdft",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f1c15276-ff93-4320-8df5-060f8f5649ca"} 1.715085514e+09
kube_pod_container_state_started{container="kube-controller-manager",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-controller-manager",pod="kube-controller-manager-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f9fcfc9a-0ca2-482d-96dd-0b5dc62dd8f2"} 1.715085477e+09
kube_pod_container_state_started{container="kube-controller-manager-cert-syncer",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-controller-manager",pod="kube-controller-manager-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f9fcfc9a-0ca2-482d-96dd-0b5dc62dd8f2"} 1.715085477e+09
kube_pod_container_state_started{container="kube-controller-manager-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-controller-manager-operator",pod="kube-controller-manager-operator-598576bd8b-lzllb",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="419b2713-4c69-4aee-ab01-d7ab786a2630"} 1.715085518e+09
kube_pod_container_state_started{container="kube-controller-manager-recovery-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-controller-manager",pod="kube-controller-manager-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f9fcfc9a-0ca2-482d-96dd-0b5dc62dd8f2"} 1.715085478e+09
kube_pod_container_state_started{container="kube-multus",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-multus",pod="multus-rrgxr",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="01ce3b9b-b1d9-4b55-9876-655ffcbd9d7f"} 1.715085496e+09
kube_pod_container_state_started{container="kube-multus-additional-cni-plugins",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-multus",pod="multus-additional-cni-plugins-8286m",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="3c5c2a60-91a3-405d-b790-e77b05b2788b"} 1.715085504e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-cluster-machine-approver",pod="machine-approver-84c96cb9bc-sqhr9",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a69c6925-e8c0-4664-a451-14d1bbd083fa"} 1.715085496e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-dns",pod="dns-default-cpdsn",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9a5f8f86-ae78-4b1b-91c1-ad298572ee95"} 1.715085506e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-dns-operator",pod="dns-operator-5d8b885cc7-k9sxq",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="b27dc45e-8b72-4757-a36c-5d3cab233853"} 1.71508551e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-ingress-operator",pod="ingress-operator-8446b9df4c-5jm72",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="85236644-7617-4ab3-bf2d-6e0761091855"} 1.715085509e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-operator-79bb575c89-fphrb",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9b6f26ff-8427-4aec-8898-aa94e48b4c1d"} 1.715085513e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-controller-54495dc9dc-nqwtk",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="210840c1-2143-4d25-8a10-28f7b58e9815"} 1.715085507e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-daemon-6xmcc",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="12caba4c-9396-4dd2-ad6f-ed952e430ba4"} 1.715085496e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-operator-6dc5bf7786-7pw7d",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="1c9dfb1f-a33d-4630-8f2c-68b892158d26"} 1.715085505e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="alertmanager-main-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4787842f-88ad-4d42-9611-8939f3adde8c"} 1.715085998e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="node-exporter-d92tw",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4e2ff01a-bdbe-4ffa-8be1-c89037a72d98"} 1.715085954e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-k8s-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6892a658-35f7-4c6f-8735-fee9163205ae"} 1.715085995e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-operator-5d6d65fbb8-b7zdl",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2bc35f35-822b-4b8c-b28c-8052a59f8601"} 1.715085941e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="telemeter-client-5d9d498f5c-wjtcx",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="290dd4c0-ba49-4323-b746-5707bc749d1c"} 1.715085978e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="thanos-querier-6cc8b8fdc7-6xnbz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="8e499d6b-9280-4223-b19c-1c547b662a26"} 1.715085981e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-multus",pod="multus-admission-controller-99b8fccfc-kk4k6",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4ba695e0-e4d4-4a89-9023-603110469cf8"} 1.715085507e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-multus",pod="network-metrics-daemon-vfk76",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="028c0c91-e715-4629-b7fe-8cf5cdacb764"} 1.715085505e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-opentelemetry-operator",pod="opentelemetry-operator-controller-manager-5bc6f98fd9-v47nq",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c3fdc2d2-ca99-4fd0-b479-67fc261c97dc"} 1.715092354e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-sdn",pod="sdn-controller-lwtkv",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="657d8cb0-f1a1-40ce-9dd6-0d9c9e490d34"} 1.715085497e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-sdn",pod="sdn-djp4v",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4ec3a54e-b906-48c8-9b5f-3ae14044aaf1"} 1.715085496e+09
kube_pod_container_state_started{container="kube-rbac-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-operator-bc7446474-khkrf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="53f00fd8-db2d-4e25-8515-49cea091a7d2"} 1.715085948e+09
kube_pod_container_state_started{container="kube-rbac-proxy-federate",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81a65bf1-d96e-41a7-b624-c88b6e0e0767"} 1.715085994e+09
kube_pod_container_state_started{container="kube-rbac-proxy-machine-mtrc",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085517e+09
kube_pod_container_state_started{container="kube-rbac-proxy-machineset-mtrc",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085516e+09
kube_pod_container_state_started{container="kube-rbac-proxy-main",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="kube-state-metrics-c7c8f4657-d869p",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="ba736941-5ad4-4984-a5d9-324ca2669aac"} 1.715085955e+09
kube_pod_container_state_started{container="kube-rbac-proxy-main",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="openshift-state-metrics-7f8ff767bd-gmtm5",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="cded0f4a-bac4-40bf-a37c-89cf55cd5762"} 1.715085948e+09
kube_pod_container_state_started{container="kube-rbac-proxy-metric",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="alertmanager-main-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4787842f-88ad-4d42-9611-8939f3adde8c"} 1.715085998e+09
kube_pod_container_state_started{container="kube-rbac-proxy-metrics",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="thanos-querier-6cc8b8fdc7-6xnbz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="8e499d6b-9280-4223-b19c-1c547b662a26"} 1.715085994e+09
kube_pod_container_state_started{container="kube-rbac-proxy-metrics",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81a65bf1-d96e-41a7-b624-c88b6e0e0767"} 1.715085994e+09
kube_pod_container_state_started{container="kube-rbac-proxy-metrics",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="thanos-ruler-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="98403218-82a1-424a-8eb1-561ff3f96b0d"} 1.715085983e+09
kube_pod_container_state_started{container="kube-rbac-proxy-mhc-mtrc",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085518e+09
kube_pod_container_state_started{container="kube-rbac-proxy-rules",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="thanos-querier-6cc8b8fdc7-6xnbz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="8e499d6b-9280-4223-b19c-1c547b662a26"} 1.715085994e+09
kube_pod_container_state_started{container="kube-rbac-proxy-self",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="kube-state-metrics-c7c8f4657-d869p",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="ba736941-5ad4-4984-a5d9-324ca2669aac"} 1.715085955e+09
kube_pod_container_state_started{container="kube-rbac-proxy-self",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="openshift-state-metrics-7f8ff767bd-gmtm5",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="cded0f4a-bac4-40bf-a37c-89cf55cd5762"} 1.715085948e+09
kube_pod_container_state_started{container="kube-rbac-proxy-thanos",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-k8s-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6892a658-35f7-4c6f-8735-fee9163205ae"} 1.715085995e+09
kube_pod_container_state_started{container="kube-rbac-proxy-thanos",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81a65bf1-d96e-41a7-b624-c88b6e0e0767"} 1.715085994e+09
kube_pod_container_state_started{container="kube-scheduler",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-scheduler",pod="openshift-kube-scheduler-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="5e7631d1-950b-44fe-96ae-bd72695a5bf5"} 1.715085477e+09
kube_pod_container_state_started{container="kube-scheduler-cert-syncer",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-scheduler",pod="openshift-kube-scheduler-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="5e7631d1-950b-44fe-96ae-bd72695a5bf5"} 1.715085477e+09
kube_pod_container_state_started{container="kube-scheduler-operator-container",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-scheduler-operator",pod="openshift-kube-scheduler-operator-6699dbdc6c-rt7hm",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="be266483-aa88-4b65-a469-770ccbd863b8"} 1.715085516e+09
kube_pod_container_state_started{container="kube-scheduler-recovery-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-scheduler",pod="openshift-kube-scheduler-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="5e7631d1-950b-44fe-96ae-bd72695a5bf5"} 1.715085478e+09
kube_pod_container_state_started{container="kube-state-metrics",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="kube-state-metrics-c7c8f4657-d869p",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="ba736941-5ad4-4984-a5d9-324ca2669aac"} 1.715085954e+09
kube_pod_container_state_started{container="kube-storage-version-migrator-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-storage-version-migrator-operator",pod="kube-storage-version-migrator-operator-85b4958b99-l56rz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c95835f3-e03e-4cef-b366-69f611680c62"} 1.715085505e+09
kube_pod_container_state_started{container="liveness-probe",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="hostpath-provisioner",pod="csi-hostpathplugin-vqcc7",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="e2fc6603-94ba-46e8-99c8-c2ad4eab7202"} 1.715085516e+09
kube_pod_container_state_started{container="machine-api-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-operator-79bb575c89-fphrb",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9b6f26ff-8427-4aec-8898-aa94e48b4c1d"} 1.715085515e+09
kube_pod_container_state_started{container="machine-approver-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-cluster-machine-approver",pod="machine-approver-84c96cb9bc-sqhr9",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a69c6925-e8c0-4664-a451-14d1bbd083fa"} 1.715085497e+09
kube_pod_container_state_started{container="machine-config-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-controller-54495dc9dc-nqwtk",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="210840c1-2143-4d25-8a10-28f7b58e9815"} 1.715085506e+09
kube_pod_container_state_started{container="machine-config-daemon",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-daemon-6xmcc",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="12caba4c-9396-4dd2-ad6f-ed952e430ba4"} 1.715085495e+09
kube_pod_container_state_started{container="machine-config-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-operator-6dc5bf7786-7pw7d",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="1c9dfb1f-a33d-4630-8f2c-68b892158d26"} 1.715085505e+09
kube_pod_container_state_started{container="machine-config-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-config-operator",pod="machine-config-server-vrwdt",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6355fe8c-bbf2-4829-a86c-305616194658"} 1.715085496e+09
kube_pod_container_state_started{container="machine-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085512e+09
kube_pod_container_state_started{container="machine-healthcheck-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085515e+09
kube_pod_container_state_started{container="machineset-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085512e+09
kube_pod_container_state_started{container="manager",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-opentelemetry-operator",pod="opentelemetry-operator-controller-manager-5bc6f98fd9-v47nq",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c3fdc2d2-ca99-4fd0-b479-67fc261c97dc"} 1.715092319e+09
kube_pod_container_state_started{container="marketplace-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="marketplace-operator-d459d8894-gpjnv",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="9e40a0b3-bc08-44e1-9d97-4f1df47981ad"} 1.715085508e+09
kube_pod_container_state_started{container="migrator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-storage-version-migrator",pod="migrator-64c5f6f5b5-tfgqw",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="b5471019-35e7-4171-9600-292804ababeb"} 1.715085514e+09
kube_pod_container_state_started{container="monitoring-plugin",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="monitoring-plugin-6c4cf96f45-9trtf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2995791a-550c-4081-84fb-cf28d405c01f"} 1.71508597e+09
kube_pod_container_state_started{container="multus-admission-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-multus",pod="multus-admission-controller-99b8fccfc-kk4k6",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4ba695e0-e4d4-4a89-9023-603110469cf8"} 1.715085506e+09
kube_pod_container_state_started{container="network-check-target-container",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-network-diagnostics",pod="network-check-target-fv5hq",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="0fdc417b-cd93-4c99-bbe3-e314f69c6cd9"} 1.715085505e+09
kube_pod_container_state_started{container="network-metrics-daemon",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-multus",pod="network-metrics-daemon-vfk76",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="028c0c91-e715-4629-b7fe-8cf5cdacb764"} 1.715085504e+09
kube_pod_container_state_started{container="network-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-network-operator",pod="network-operator-699984f98c-47sqx",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="1eb16091-343a-402e-b160-695e373b1f8c"} 1.715085496e+09
kube_pod_container_state_started{container="node-ca",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-image-registry",pod="node-ca-9x6cc",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="d5ee5d6c-ef81-4da1-97c7-056998dfdb37"} 1.715085496e+09
kube_pod_container_state_started{container="node-driver-registrar",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="hostpath-provisioner",pod="csi-hostpathplugin-vqcc7",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="e2fc6603-94ba-46e8-99c8-c2ad4eab7202"} 1.715085514e+09
kube_pod_container_state_started{container="node-exporter",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="node-exporter-d92tw",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4e2ff01a-bdbe-4ffa-8be1-c89037a72d98"} 1.715085954e+09
kube_pod_container_state_started{container="nodelink-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-machine-api",pod="machine-api-controllers-7d597f4dbd-wknbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a0428261-f6aa-40f7-a05d-3d0e40ce0901"} 1.715085513e+09
kube_pod_container_state_started{container="oauth-apiserver",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-oauth-apiserver",pod="apiserver-696fccdfbc-llg9c",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f7936908-cbf0-402c-bd79-f95adf00872c"} 1.715085516e+09
kube_pod_container_state_started{container="oauth-openshift",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-authentication",pod="oauth-openshift-7df4bd8d4c-mpdjj",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="52ab8607-6d49-4d8d-b58a-95e1d8863749"} 1.715085559e+09
kube_pod_container_state_started{container="oauth-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="thanos-querier-6cc8b8fdc7-6xnbz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="8e499d6b-9280-4223-b19c-1c547b662a26"} 1.715085981e+09
kube_pod_container_state_started{container="olm-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="olm-operator-7c89b67546-tfz4j",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="ed87890f-7269-47e7-b295-1510e5b73077"} 1.71508551e+09
kube_pod_container_state_started{container="openshift-apiserver",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-apiserver",pod="apiserver-76dcb49f46-rds48",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2351b426-2823-4d26-bc98-33d60255411d"} 1.715085508e+09
kube_pod_container_state_started{container="openshift-apiserver-check-endpoints",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-apiserver",pod="apiserver-76dcb49f46-rds48",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2351b426-2823-4d26-bc98-33d60255411d"} 1.715085509e+09
kube_pod_container_state_started{container="openshift-apiserver-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-apiserver-operator",pod="openshift-apiserver-operator-86bf4c84dd-t6mkw",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="f2883174-7a6c-4de2-bb8d-de72422100e8"} 1.715085504e+09
kube_pod_container_state_started{container="openshift-config-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-config-operator",pod="openshift-config-operator-6b84cc88f6-hrl9s",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="3ef7f846-827a-42f3-9f9a-dc0ad3ad2aa0"} 1.715085507e+09
kube_pod_container_state_started{container="openshift-controller-manager-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-controller-manager-operator",pod="openshift-controller-manager-operator-76d9b9c7f-hrpfx",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="64705345-51a9-48da-9edc-36c8b9ebe5eb"} 1.715085513e+09
kube_pod_container_state_started{container="openshift-state-metrics",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="openshift-state-metrics-7f8ff767bd-gmtm5",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="cded0f4a-bac4-40bf-a37c-89cf55cd5762"} 1.715085971e+09
kube_pod_container_state_started{container="otc-container",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="observability",pod="otel-collector-6f64d649c4-h5dvl",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="67579aeb-5a5f-44c7-8821-71f8a8cf8b81"} 1.715097926e+09
kube_pod_container_state_started{container="otc-container",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="observability",pod="otel-collector-6f64d649c4-x9p9x",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c8512bcd-20cd-4bb7-93d1-8bf4101cd108"} 1.715098034e+09
kube_pod_container_state_started{container="package-server-manager",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="package-server-manager-84dd4fc4b7-k9hr8",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="370aa8e0-441f-4c5e-b53d-b13068144c12"} 1.715085505e+09
kube_pod_container_state_started{container="packageserver",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-operator-lifecycle-manager",pod="packageserver-654dbbdb7-rd5cp",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="befd0731-4817-4800-9c7f-7f3a58e8976f"} 1.715085506e+09
kube_pod_container_state_started{container="prom-label-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="alertmanager-main-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4787842f-88ad-4d42-9611-8939f3adde8c"} 1.715085998e+09
kube_pod_container_state_started{container="prom-label-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="thanos-querier-6cc8b8fdc7-6xnbz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="8e499d6b-9280-4223-b19c-1c547b662a26"} 1.715085993e+09
kube_pod_container_state_started{container="prometheus",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-k8s-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6892a658-35f7-4c6f-8735-fee9163205ae"} 1.715085994e+09
kube_pod_container_state_started{container="prometheus",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81a65bf1-d96e-41a7-b624-c88b6e0e0767"} 1.715085992e+09
kube_pod_container_state_started{container="prometheus-adapter",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-adapter-7956759cc-lxss4",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="24562bfe-c410-4760-b24b-8af214ee63a0"} 1.715085979e+09
kube_pod_container_state_started{container="prometheus-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-operator-5d6d65fbb8-b7zdl",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="2bc35f35-822b-4b8c-b28c-8052a59f8601"} 1.715085941e+09
kube_pod_container_state_started{container="prometheus-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-operator-bc7446474-khkrf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="53f00fd8-db2d-4e25-8515-49cea091a7d2"} 1.715085948e+09
kube_pod_container_state_started{container="prometheus-operator-admission-webhook",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-operator-admission-webhook-6c9c7ff8dd-99tgf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="a8c155a7-e78c-4894-a73c-7df467f52d88"} 1.715085921e+09
kube_pod_container_state_started{container="prometheus-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-k8s-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6892a658-35f7-4c6f-8735-fee9163205ae"} 1.715085995e+09
kube_pod_container_state_started{container="pruner",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-kube-scheduler",pod="revision-pruner-8-crc-ksq4m-master-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="217506b4-ee0c-4a86-a224-34689a2eb2a9"} 1.715085508e+09
kube_pod_container_state_started{container="registry",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-image-registry",pod="image-registry-85cd4b6467-7kxdp",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="0944dffc-2a84-4dbb-9f5e-e8e09ccd3492"} 1.715085531e+09
kube_pod_container_state_started{container="registry-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="certified-operators-r8vmw",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81058249-9471-4810-994e-01516b56ffa8"} 1.715085778e+09
kube_pod_container_state_started{container="registry-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="community-operators-57wsh",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="03f543a1-6de0-471d-8e81-bc631bc7acd3"} 1.715095183e+09
kube_pod_container_state_started{container="registry-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="redhat-marketplace-l2x4q",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="ee4048d3-cdda-4d3f-b5a7-cdd455fdf405"} 1.715085803e+09
kube_pod_container_state_started{container="registry-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="redhat-operators-phmfg",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c3e06347-a550-4e9a-9f56-f00bcee60d3c"} 1.715098179e+09
kube_pod_container_state_started{container="registry-server",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-marketplace",pod="redhat-operators-wgrmj",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6371f933-9505-4a05-b411-5adaf49249b0"} 1.715085818e+09
kube_pod_container_state_started{container="reload",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="telemeter-client-5d9d498f5c-wjtcx",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="290dd4c0-ba49-4323-b746-5707bc749d1c"} 1.715085977e+09
kube_pod_container_state_started{container="route-controller-manager",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-route-controller-manager",pod="route-controller-manager-584cdb6f9f-cl959",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c25964a3-df01-423e-9e4a-6d1fe0a5a159"} 1.715085561e+09
kube_pod_container_state_started{container="router",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-ingress",pod="router-default-7bc45ddd7d-w587s",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="26be0eb6-5080-48df-b23e-e8f1bd250437"} 1.715085495e+09
kube_pod_container_state_started{container="sdn",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-sdn",pod="sdn-djp4v",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4ec3a54e-b906-48c8-9b5f-3ae14044aaf1"} 1.715085496e+09
kube_pod_container_state_started{container="sdn-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-sdn",pod="sdn-controller-lwtkv",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="657d8cb0-f1a1-40ce-9dd6-0d9c9e490d34"} 1.715085496e+09
kube_pod_container_state_started{container="serve-healthcheck-canary",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-ingress-canary",pod="ingress-canary-6mn77",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="c42f5fc6-deb9-49ab-9ccb-192ca60330dc"} 1.715085508e+09
kube_pod_container_state_started{container="service-ca-controller",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-service-ca",pod="service-ca-585c8dc78-9bbw5",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="fe89c67f-4303-49d5-a8a0-ab02bb56b112"} 1.715085518e+09
kube_pod_container_state_started{container="service-ca-operator",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-service-ca-operator",pod="service-ca-operator-6bf4db99cf-94sbs",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="4b3904fc-963e-4c55-a708-9289c9d21466"} 1.71508551e+09
kube_pod_container_state_started{container="telemeter-client",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="telemeter-client-5d9d498f5c-wjtcx",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="290dd4c0-ba49-4323-b746-5707bc749d1c"} 1.715085977e+09
kube_pod_container_state_started{container="thanos-query",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="thanos-querier-6cc8b8fdc7-6xnbz",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="8e499d6b-9280-4223-b19c-1c547b662a26"} 1.71508597e+09
kube_pod_container_state_started{container="thanos-ruler",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="thanos-ruler-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="98403218-82a1-424a-8eb1-561ff3f96b0d"} 1.715085983e+09
kube_pod_container_state_started{container="thanos-ruler-proxy",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="thanos-ruler-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="98403218-82a1-424a-8eb1-561ff3f96b0d"} 1.715085983e+09
kube_pod_container_state_started{container="thanos-sidecar",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-monitoring",pod="prometheus-k8s-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="6892a658-35f7-4c6f-8735-fee9163205ae"} 1.715085994e+09
kube_pod_container_state_started{container="thanos-sidecar",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-user-workload-monitoring",pod="prometheus-user-workload-0",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state-metrics",uid="81a65bf1-d96e-41a7-b624-c88b6e0e0767"} 1.715085993e+09
kube_pod_container_state_started{container="webhook",endpoint="https-main",exported_job="kube-state-metrics",instance="prometheus-k8s.openshift-monitoring.svc.cluster.local:9091",job="federate",namespace="openshift-network-node-identity",pod="network-node-identity-qjmkf",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0",service="kube-state

collector logs

2024-05-07T16:06:22.994Z	info	service@v0.93.0/telemetry.go:76	Setting up own telemetry...
2024-05-07T16:06:22.994Z	info	service@v0.93.0/telemetry.go:146	Serving metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2024-05-07T16:06:22.996Z	info	service@v0.93.0/service.go:139	Starting otelcol...	{"Version": "0.93.0", "NumCPU": 5}
2024-05-07T16:06:22.996Z	info	extensions/extensions.go:34	Starting extensions...
2024-05-07T16:06:22.996Z	warn	internal@v0.93.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-05-07T16:06:22.996Z	warn	internal@v0.93.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-05-07T16:06:22.996Z	info	otlpreceiver@v0.93.0/otlp.go:102	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4317"}
2024-05-07T16:06:22.997Z	warn	internal@v0.93.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-05-07T16:06:22.997Z	info	otlpreceiver@v0.93.0/otlp.go:152	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
2024-05-07T16:06:22.997Z	info	prometheusreceiver@v0.93.0/metrics_receiver.go:240	Starting discovery manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-05-07T16:06:22.998Z	info	prometheusreceiver@v0.93.0/metrics_receiver.go:231	Scrape job added	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "federate"}
2024-05-07T16:06:22.999Z	info	service@v0.93.0/service.go:165	Everything is ready. Begin running and processing data.
2024-05-07T16:06:22.999Z	info	prometheusreceiver@v0.93.0/metrics_receiver.go:282	Starting scrape manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-05-07T16:06:32.887Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715097992843, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:06:37.861Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715097997847, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:06:43.174Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715098003165, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:06:47.854Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715098007846, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:06:52.859Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715098012850, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:06:57.851Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715098017846, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:07:02.852Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715098022846, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}
2024-05-07T16:07:07.853Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1715098027846, "target_labels": "{__name__=\"up\", instance=\"prometheus-k8s.openshift-monitoring.svc.cluster.local:9091\", job=\"federate\"}"}