Description
Component(s)
pkg/stanza
What happened?
Description
When running with an udplogreceiver -> batch processor -> exporter with very high load for long periods of time we are seeing an occasional panic (see stack below)
I believe the source of the problem is here:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/adapter/receiver.go#L128
Line 128 calls pLogs.LogRecordCount after line 124 calls r.consumer.ConsumeLogs.
Issue 1: If the consumer is mutating the pLogs asynchronously while LogRecordCount is looping, you can get a panic.
Issue 2: If pLogs is modified synchronously in ConsumeLogs, the count could be wrong (assuming that the incoming count is what is desired)
Proposed solution:
I think the count should be taken before ConsumeLogs is called and saved in a local var for EndLogsOp to use later.
Steps to Reproduce
Expected Result
Actual Result
Collector version
v0.88.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
No response
Log output
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xac4840]
goroutine 182 [running]:
go.opentelemetry.io/collector/pdata/plog.ResourceLogsSlice.At(...)
go.opentelemetry.io/collector/[email protected]/plog/generated_resourcelogsslice.go:56
go.opentelemetry.io/collector/pdata/plog.Logs.LogRecordCount({0xc002760678?, 0xc001f3cc58?})
go.opentelemetry.io/collector/[email protected]/plog/logs.go:48 +0x20
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).consumerLoop(0xc000591950, {0x3ecf9b8, 0xc000895860})
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/receiver.go:128 +0x325
created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).Start in goroutine 1
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/adapter/receiver.go:68 +0x285Additional context
No response