Splunk otel collector own metrics?

[tomsucho]

Hi,
does splunk otel collector components expose their own metrics? If so could you please point me to the relevant configuration in Helm chart how to get it enabled? I would be specifically interested in any metrics that could tell me if the log shipping to Splunk Cloud goes fine, so would appreciate any hints on that.

[hgus-gushernandez]

At this time, it is not possible to collect metrics from the OTEL components.
However, assuming there are no configuration or communication issues which can impact the agent functionality, the assumption is that all the logs should make it to the destination.
In the event that you do not see logs being forwarded by the agent, you can use the agent logs to investigate any issues.
If necessary, you can enable debug mode to gather more detailed logs that can assist in your investigation.
Lastly, if you run into any technical difficulties with the above, I recommend you file a Support Case with Splunk at http://login.splunk.com/ for assistance to investigate such issues.

Below are links to documentation you may find useful.

Troubleshoot the Splunk OpenTelemetry Collector
https://docs.splunk.com/Observability/gdi/opentelemetry/splunk-collector-troubleshooting.html

Troubleshooting
https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md

[tomsucho]

okay I understand, could you please make a feature like that? I think it would be quite useful. I know you could probably get something setup from Splunk Cloud side to get alerted, but from the agent side it would be nice as well I think i.e. if volume of data/logs suddenly drops, timeouts, bottlenecks or errors occur impacting data shipment, it would be easy to build monitoring & alerting using well known Prometheus stack based on such metrics.
EDIT: I know there are these gateway pods so I can try to maybe get something working based on low level container network bytes tx metrics maybe.

[hgus-gushernandez]

The best thing to do is to submit the idea at https://ideas.splunk.com

Thank You

[tomsucho]

@hgus-gushernandez I also found this: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/monitoring.md
so it would look like there are some metrics built in but maybe just not exposed on any port/endpoint yet in the Helm chart?

[hgus-gushernandez]

The Metrics or this document is specific to Splunk Observability and not Splunk Cloud
https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/monitoring.md

You can read more about each product in the links I have added for each product

[tomsucho]

do you have any better links docs? those links seems to me like high level marketing stuff. Isn't this repo stuff based on open telemetry collector? it has it even in the name? It would look like to me those are maybe two different Cloud backends with different capabilities maybe, but the agent/client side should be pretty much common and based on open telemetry collector which is oss?

[hgus-gushernandez]

Unfortunately, no.
They are different backends with different capabilities and the Splunk OTEL can be used to send data to either back end.
I would recommend you sign up for the free Splunk Observability trial and experiment with the Splunk OTEL in Observability to see the functionality first hand.
During your trial, you will be engaged with a Splunk Observability expert who can address all your concerns regarding Opentelemetry.

[tomsucho]

@hgus-gushernandez so I actually logged into one of the DaemonSet splunk-otel-collector-agent pods and was able to find out the otelcol process actually has the port 8889 opened and appears to be talking http and guess what? it even responds with metrics.. btw the port 24321 is not there, even tho it is in container spec and named http-metrics...
But okay I think I can overlay on top of the helm chart manifests to add that 8889 port to the container and then get it scraped by Prometheus.. But you guys should be exposing this and I should not be needing to hack this to find out this simple info...

/ # telnet localhost 8889
Connected to localhost
GET /metrics HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Fri, 26 May 2023 16:47:00 GMT
Transfer-Encoding: chunked

800
# HELP otelcol_exporter_enqueue_failed_log_records Number of log records failed to be added to the sending queue.
# TYPE otelcol_exporter_enqueue_failed_log_records counter
(...)

[atoulme]

We scrape this endpoint as part of our deployment, please see here:

splunk-otel-collector-chart/helm-charts/splunk-otel-collector/templates/config/_otel-agent.tpl

Line 40 in ea3c4f9

- "${K8S_POD_IP}:8889"

[tomsucho]

@atoulme okay, but where's this port enabled on the agent DaemonSet? On my Helm inflated manifests I don't see 8889 opened on containers anywhere, and looking in the chart values file I don't seem to find 8889 here?

splunk-otel-collector-chart/helm-charts/splunk-otel-collector/values.yaml

Lines 313 to 356 in ea3c4f9

	ports:
	otlp:
	containerPort: 4317
	hostPort: 4317
	protocol: TCP
	enabled_for: [traces, metrics, logs, profiling]
	otlp-http:
	containerPort: 4318
	protocol: TCP
	enabled_for: [metrics, traces, logs, profiling]
	otlp-http-old:
	containerPort: 55681
	protocol: TCP
	enabled_for: [metrics, traces, logs]
	sfx-forwarder:
	containerPort: 9080
	hostPort: 9080
	protocol: TCP
	enabled_for: [traces]
	zipkin:
	containerPort: 9411
	hostPort: 9411
	protocol: TCP
	enabled_for: [traces]
	jaeger-thrift:
	containerPort: 14268
	hostPort: 14268
	protocol: TCP
	enabled_for: [traces]
	jaeger-grpc:
	containerPort: 14250
	hostPort: 14250
	protocol: TCP
	enabled_for: [traces]
	fluentforward:
	containerPort: 8006
	hostPort: 8006
	protocol: TCP
	enabled_for: [logs]
	signalfx:
	containerPort: 9943
	hostPort: 9943
	protocol: TCP
	enabled_for: [metrics]

I'm going to patch it to add on top like this:

    name: otel-collector
    ports:
    - containerPort: 8889
      name: http-metrics

[jvoravong]

The port wasn't added to the daemonset spec. I believe this was intentionally done because we only wanted the agent collecting it's own metrics.

• splunk-otel-collector-chart/examples/default/rendered_manifests/configmap-agent.yaml

  Line 255 in 7698ca2

  metrics/agent:

• splunk-otel-collector-chart/examples/default/rendered_manifests/configmap-agent.yaml

  Line 179 in 7698ca2

  - ${K8S_POD_IP}:8889

• splunk-otel-collector-chart/examples/default/rendered_manifests/configmap-agent.yaml

  Line 283 in 7698ca2

  address: 0.0.0.0:8889

[tomsucho]

@jvoravong but how do I then get those metrics into my Promethus in the cluster to alert me on say:

otelcol_exporter_send_failed_log_records

?
btw. I patched the ds & deployments to expose port 8889 and getting it scraped now, so are you saying I will now get all the metrics that go into Splunk Cloud also in local cluster Prometheus instance?

[atoulme]

We do not support exporting to Prometheus at this time. You can send to Splunk Enterprise or Splunk Observability Cloud.

[tomsucho]

okay, but this doesn't sound right i.e. why would I look for a metric on a Splunk backend Cloud to tell me if the log/metrics shipping is borked? E.g. if sending goes wrong on client side, then on the backend side might even get nothing i.e. no logs/metrics? and while you can tell, well that's indicating clearly a problem, it doesn't give you any clue what was going on and so having the ability to monitor some key operational metrics locally in Prometheus in my view is a key function for any 'cloud native' production ready system..
For us however, since we use it only for log shipping and metric collection is disabled anyways, I enabled that port 8889 through kustomize overlay, and added scraping to Prometheus and from what I can see, from that scrape job I'm getting only otelcol_* like metrics, around 400 of them, including the one I think that is quite useful to detect send issues:

      expr: |
        rate(otelcol_exporter_send_failed_log_records[5m]) > 0
      for: 30m