Description
Component(s)
TargetAllocator
What happened?
Description
When the service/pod monitors use basic auth and read the keys from secrets, it is able to get them using the new https support. However, when the secret gets updated, the pod/service monitors do not pick them up.
Another related issue is that is the pod/service monitor is created before the secret object is created, the pod/svc monitors don't pick them up and makes it dependent on the order of resource creation.
Steps to Reproduce
Create pod/service monitor using basic auth reading the username and password from secret. Update the username/password in the secret. But the pod and service monitors don't pick them up.
Create secret being referenced in the service monitor after the service monitor is created. The secret doesn't get picked up and the pod and service monitors do not work as expected.
Expected Result
When the secret is updated the new values should be updated in the objStore which the pod/service monitors use.
The order of secret creation shouldn't affect pod/svc monitor evaluation, should get picked up whenever the secret is created.
Actual Result
Updated secret values don't get picked up from pod/svc monitors.
If secret is created after pod/svc monitor creation, it gets skipped.
Kubernetes Version
1.29.15
Operator version
v0.127.0
Collector version
v0.127.0
Environment information
Environment
OS: cbl-mariner - 2.0
Compiler(if manually compiled): 1.23.8
Log output
Example service monitor for the second part of the issue.
endpoints:
- basicAuth:
password:
key: password
name: my-secret-with-basic-auth-creds
username:
key: username
name: my-secret-with-basic-auth-creds
{"level":"info","ts":"2025-06-10T19:32:26Z","msg":"Event occurred","object":{"name":"svc-prometheus-reference-app-basic-auth-1","namespace":"default"},"fieldPath":"","kind":"ServiceMonitor","apiVersion":"azmonitoring.coreos.com/v1","type":"Warning","reason":"InvalidConfiguration","message":"ServiceMonitor svc-prometheus-reference-app-basic-auth-1 was rejected due to invalid configuration: failed to get basic auth username: unable to get secret \"my-secret-with-basic-auth-creds\": secrets \"my-secret-with-basic-auth-creds\" not found"}Additional context
The fix should be to add a secret informer to the list of existing informers
and update objStore on secret update event
Currently prometheus-operator doesnt provide ability to update objStore from package consumers, created an issue and PR to address this. Once that merges, happy to make the PR here.