Add log support to Splunk HEC exporter #875
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #875 +/- ##
==========================================
+ Coverage 88.86% 89.09% +0.23%
==========================================
Files 272 261 -11
Lines 13294 12520 -774
==========================================
- Hits 11814 11155 -659
+ Misses 1125 1014 -111
+ Partials 355 351 -4
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
atoulme
force-pushed
the
splunkhecexporter_logs
branch
2 times, most recently
from
426076f to
54f2054
Compare
atoulme
force-pushed
the
splunkhecexporter_logs
branch
2 times, most recently
from
80abacb to
de17234
Compare
keitwb
reviewed
Sep 1, 2020
keitwb
reviewed
Sep 1, 2020
keitwb
reviewed
Sep 1, 2020
keitwb
reviewed
Sep 1, 2020
atoulme
force-pushed
the
splunkhecexporter_logs
branch
from
de17234 to
9f6af1c
Compare
keitwb
reviewed
Sep 1, 2020
keitwb
reviewed
Sep 1, 2020
internal/common/splunk/common.go
Outdated
| @@ -17,8 +17,11 @@ package splunk | |||
| import "strings" | |||
|
|
|||
| const ( | |||
| HostnameLabel = "host.hostname" | |||
There was a problem hiding this comment.
I just realized these are the same as in this file https://github.com/open-telemetry/opentelemetry-collector/blob/master/translator/conventions/opentelemetry.go#L20. If they are actually referring to the same thing then you can just refer to that package instead of redefining them here. Looks like source.type isn't there but the other two are.
|
I thought I could push to this to update it but I can't. Anyway I made some updates for when you get back from leave: diff --git a/exporter/splunkhecexporter/client_test.go b/exporter/splunkhecexporter/client_test.go
index d736ccb..d1c0431 100644
--- a/exporter/splunkhecexporter/client_test.go
+++ b/exporter/splunkhecexporter/client_test.go
@@ -36,6 +36,7 @@ import (
"go.opentelemetry.io/collector/consumer/pdata"
"go.opentelemetry.io/collector/consumer/pdatautil"
"go.opentelemetry.io/collector/testutil/metricstestutil"
+ "go.opentelemetry.io/collector/translator/conventions"
"go.opentelemetry.io/collector/translator/internaldata"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/timestamppb"
@@ -118,9 +119,9 @@ func createLogData(numberOfLogs int) pdata.Logs {
logRecord := pdata.NewLogRecord()
logRecord.InitEmpty()
logRecord.Body().SetStringVal("mylog")
- logRecord.Attributes().InsertString(splunk.SourceLabel, "myapp")
+ logRecord.Attributes().InsertString(conventions.AttributeServiceName, "myapp")
logRecord.Attributes().InsertString(splunk.SourcetypeLabel, "myapp-type")
- logRecord.Attributes().InsertString(splunk.HostnameLabel, "myhost")
+ logRecord.Attributes().InsertString(conventions.AttributeHostHostname, "myhost")
logRecord.Attributes().InsertString("custom", "custom")
logRecord.SetTimestamp(ts)
ill.Logs().Append(&logRecord)
diff --git a/exporter/splunkhecexporter/exporter_test.go b/exporter/splunkhecexporter/exporter_test.go
index 7291399..a48f7d0 100644
--- a/exporter/splunkhecexporter/exporter_test.go
+++ b/exporter/splunkhecexporter/exporter_test.go
@@ -36,6 +36,7 @@ import (
"go.opentelemetry.io/collector/consumer/pdata"
"go.opentelemetry.io/collector/consumer/pdatautil"
"go.opentelemetry.io/collector/testutil/metricstestutil"
+ "go.opentelemetry.io/collector/translator/conventions"
"go.uber.org/zap"
"github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/splunk"
@@ -205,9 +206,9 @@ func generateLargeLogsBatch(t *testing.T) pdata.Logs {
logRecord := pdata.NewLogRecord()
logRecord.InitEmpty()
logRecord.Body().SetStringVal("mylog")
- logRecord.Attributes().InsertString(splunk.SourceLabel, "myapp")
+ logRecord.Attributes().InsertString(conventions.AttributeServiceName, "myapp")
logRecord.Attributes().InsertString(splunk.SourcetypeLabel, "myapp-type")
- logRecord.Attributes().InsertString(splunk.HostnameLabel, "myhost")
+ logRecord.Attributes().InsertString(conventions.AttributeHostHostname, "myhost")
logRecord.Attributes().InsertString("custom", "custom")
logRecord.SetTimestamp(ts)
}
@@ -219,7 +220,7 @@ func TestConsumeLogsData(t *testing.T) {
logRecord := pdata.NewLogRecord()
logRecord.InitEmpty()
logRecord.Body().SetStringVal("mylog")
- logRecord.Attributes().InsertString(splunk.HostnameLabel, "myhost")
+ logRecord.Attributes().InsertString(conventions.AttributeHostHostname, "myhost")
logRecord.Attributes().InsertString("custom", "custom")
logRecord.SetTimestamp(123)
smallBatch := makeLog(logRecord)
diff --git a/exporter/splunkhecexporter/logdata_to_splunk.go b/exporter/splunkhecexporter/logdata_to_splunk.go
index ba1ddae..3ab201c 100644
--- a/exporter/splunkhecexporter/logdata_to_splunk.go
+++ b/exporter/splunkhecexporter/logdata_to_splunk.go
@@ -15,9 +15,10 @@
package splunkhecexporter
import (
- "math"
+ "time"
"go.opentelemetry.io/collector/consumer/pdata"
+ "go.opentelemetry.io/collector/translator/conventions"
"go.uber.org/zap"
"github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/splunk"
@@ -68,9 +69,9 @@ func mapLogRecordToSplunkEvent(lr pdata.LogRecord, config *Config) *splunkEvent
var sourcetype string
fields := map[string]string{}
lr.Attributes().ForEach(func(k string, v pdata.AttributeValue) {
- if k == splunk.HostnameLabel {
+ if k == conventions.AttributeHostHostname {
host = v.StringVal()
- } else if k == splunk.SourceLabel {
+ } else if k == conventions.AttributeServiceName {
source = v.StringVal()
} else if k == splunk.SourcetypeLabel {
sourcetype = v.StringVal()
@@ -104,5 +105,5 @@ func mapLogRecordToSplunkEvent(lr pdata.LogRecord, config *Config) *splunkEvent
// nanoTimestampToEpochMilliseconds transforms nanoseconds into <sec>.<ms>. For example, 1433188255.500 indicates 1433188255 seconds and 500 milliseconds after epoch.
func nanoTimestampToEpochMilliseconds(ts pdata.TimestampUnixNano) float64 {
- return float64(ts/1e9) + math.Round(float64(ts%1e9)/1e6)/1e3
+ return time.Duration(ts).Round(time.Millisecond).Seconds()
}
diff --git a/exporter/splunkhecexporter/logdata_to_splunk_test.go b/exporter/splunkhecexporter/logdata_to_splunk_test.go
index edd28d7..bf8cc61 100644
--- a/exporter/splunkhecexporter/logdata_to_splunk_test.go
+++ b/exporter/splunkhecexporter/logdata_to_splunk_test.go
@@ -20,6 +20,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/consumer/pdata"
+ "go.opentelemetry.io/collector/translator/conventions"
"go.uber.org/zap"
"github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/splunk"
@@ -42,9 +43,9 @@ func Test_logDataToSplunk(t *testing.T) {
logRecord := pdata.NewLogRecord()
logRecord.InitEmpty()
logRecord.Body().SetStringVal("mylog")
- logRecord.Attributes().InsertString(splunk.SourceLabel, "myapp")
+ logRecord.Attributes().InsertString(conventions.AttributeServiceName, "myapp")
logRecord.Attributes().InsertString(splunk.SourcetypeLabel, "myapp-type")
- logRecord.Attributes().InsertString(splunk.HostnameLabel, "myhost")
+ logRecord.Attributes().InsertString(conventions.AttributeHostHostname, "myhost")
logRecord.Attributes().InsertString("custom", "custom")
logRecord.SetTimestamp(ts)
return makeLog(logRecord)
diff --git a/exporter/splunkhecexporter/metricdata_to_splunk.go b/exporter/splunkhecexporter/metricdata_to_splunk.go
index 6ae70df..886330b 100644
--- a/exporter/splunkhecexporter/metricdata_to_splunk.go
+++ b/exporter/splunkhecexporter/metricdata_to_splunk.go
@@ -23,6 +23,7 @@ import (
metricspb "github.com/census-instrumentation/opencensus-proto/gen-go/metrics/v1"
"go.opentelemetry.io/collector/consumer/pdata"
"go.opentelemetry.io/collector/consumer/pdatautil"
+ "go.opentelemetry.io/collector/translator/conventions"
"go.uber.org/zap"
"google.golang.org/protobuf/types/known/timestamppb"
@@ -62,7 +63,7 @@ func metricDataToSplunk(logger *zap.Logger, data pdata.Metrics, config *Config)
for _, ocmd := range ocmds {
var host string
if ocmd.Resource != nil {
- host = ocmd.Resource.Labels[splunk.HostnameLabel]
+ host = ocmd.Resource.Labels[conventions.AttributeHostHostname]
}
if host == "" {
host = unknownHostName
diff --git a/exporter/splunkhecexporter/tracedata_to_splunk.go b/exporter/splunkhecexporter/tracedata_to_splunk.go
index 4322671..251deb5 100644
--- a/exporter/splunkhecexporter/tracedata_to_splunk.go
+++ b/exporter/splunkhecexporter/tracedata_to_splunk.go
@@ -16,10 +16,9 @@ package splunkhecexporter
import (
"go.opentelemetry.io/collector/consumer/pdata"
+ "go.opentelemetry.io/collector/translator/conventions"
"go.opentelemetry.io/collector/translator/internaldata"
"go.uber.org/zap"
-
- "github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/splunk"
)
type splunkEvent struct {
@@ -39,7 +38,7 @@ func traceDataToSplunk(logger *zap.Logger, data pdata.Traces, config *Config) ([
for _, octd := range octds {
var host string
if octd.Resource != nil {
- host = octd.Resource.Labels[splunk.HostnameLabel]
+ host = octd.Resource.Labels[conventions.AttributeHostHostname]
}
if host == "" {
host = unknownHostName
diff --git a/internal/common/go.sum b/internal/common/go.sum
index 680aae0..c888df6 100644
--- a/internal/common/go.sum
+++ b/internal/common/go.sum
@@ -41,6 +41,7 @@ github.com/containerd/containerd v1.3.6 h1:SMfcKoQyWhaRsYq7290ioC6XFcHDNcHvcEMjF
github.com/containerd/containerd v1.3.6/go.mod h1:bC6axHOhabU15QhwfG7w5PipXdVtMXFTttgp+kVtyUA=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
@@ -148,6 +149,7 @@ github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zM
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
@@ -161,6 +163,7 @@ github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
diff --git a/internal/common/splunk/common.go b/internal/common/splunk/common.go
index 1c340f6..1cb394f 100644
--- a/internal/common/splunk/common.go
+++ b/internal/common/splunk/common.go
@@ -17,10 +17,8 @@ package splunk
import "strings"
const (
- HostnameLabel = "host.hostname"
SFxAccessTokenHeader = "X-Sf-Token"
SFxAccessTokenLabel = "com.splunk.signalfx.access_token"
- SourceLabel = "service.name"
SourcetypeLabel = "source.type"
) |
|
🙇 thank you Ben! |
atoulme
force-pushed
the
splunkhecexporter_logs
branch
from
27c66dc to
b79006d
Compare
6 tasks
keitwb
reviewed
Sep 25, 2020
internal/common/splunk/common.go
Outdated
| @@ -21,6 +21,7 @@ const ( | |||
| SFxAccessTokenLabel = "com.splunk.signalfx.access_token" | |||
| SFxEventCategoryKey = "com.splunk.signalfx.event_category" | |||
| SFxEventPropertiesKey = "com.splunk.signalfx.event_properties" | |||
| SourcetypeLabel = "source.type" | |||
There was a problem hiding this comment.
Can we use "com.splunk.sourcetype" instead?
There was a problem hiding this comment.
In this case we cannot - "source.type" is a field that is part of the Splunk log payload. It is defined here:
https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/logs/data-model.md#splunk-hec
There was a problem hiding this comment.
I believe the definition in the spec is incorrect. I wrote it before Otel introduced guidelines for conventions which say companies need to use reverse domain name as a prefix: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/common/common.md#recommendations-for-application-developers
The name is specific to your company and may be possibly used outside the company as well. To avoid clashes with names introduced by other companies (in a distributed system that uses applications from multiple vendors) it is recommended to prefix the new name by your company's reverse domain name, e.g. com.acme.shopname.
There was a problem hiding this comment.
Makes sense, making the change.
|
@atoulme please resolve the conflict. |
tigrannajaryan
approved these changes
Sep 29, 2020
|
@atoulme Thank you. |
dyladan
referenced
this pull request
in dynatrace-oss-contrib/opentelemetry-collector-contrib
Jan 29, 2021
Signed-off-by: Bogdan Drutu <[email protected]>
codeboten
pushed a commit
that referenced
this pull request
Nov 23, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Adds log support to Splunk HEC exporter.
Testing:
Unit tests.
Documentation:
None.