Panic after upgrade from 0.126 to 0.127

[plengauer]

Component(s)

No response

What happened?

Description

I run a collector in a github action. after upgrading from 0.126 to 127, it starts panicing

Steps to Reproduce

in a github action, run this command:

sudo docker run --network=host --mount type=bind,source="$(pwd)"/collector.yaml,target=/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.127.0

Expected Result

The collector working and respecting the config as it does in 0.126

Actual Result

Panic!

Collector version

0.127

Environment information

Environment

GitHub action ubuntu-latest

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: localhost:4318
exporters:
  otlphttp/logs:
    endpoint: "http://localhost:8080"
    headers:
  otlphttp/metrics:
    endpoint: "http://localhost:8080"
    headers:
  otlphttp/traces:
    endpoint: "http://localhost:8080"
    headers:
processors:
  batch:
  transform:
    error_mode: ignore
    log_statements:
      - replace_all_patterns(log.attributes, "value", "***", "***")
      - replace_all_patterns(log.attributes, "value", "***", "***")
      - replace_all_patterns(log.attributes, "value", "***", "***")
      - replace_pattern(log.body, "***", "***")
      - replace_pattern(log.body, "***", "***")
      - replace_pattern(log.body, "***", "***")
    metric_statements:
      - replace_all_patterns(datapoint.attributes, "value", "***", "***")
      - replace_all_patterns(datapoint.attributes, "value", "***", "***")
      - replace_all_patterns(datapoint.attributes, "value", "***", "***")
    trace_statements:
      - replace_all_patterns(span.attributes, "value", "***", "***")
      - replace_all_patterns(span.attributes, "value", "***", "***")
      - replace_all_patterns(span.attributes, "value", "***", "***")
      - replace_pattern(span.name, "***", "***")
      - replace_pattern(span.name, "***", "***")
      - replace_pattern(span.name, "***", "***")
service:
  telemetry:
    metrics:
      level: none
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]
      processors: [transform, batch]
    metrics:
      receivers: [otlp]
      exporters: [otlphttp/metrics]
      processors: [transform, batch]
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
      processors: [transform, batch]

Log output

panic: reflect.Set: value of type map[string]interface {} is not assignable to type map[string]configopaque.String
goroutine 1 [running]:
reflect.Value.assignTo({0xa894480?, 0x0?, 0x146ca080?}, {0xbd540d5, 0xb}, 0xa898020, 0x0)
	reflect/value.go:3072 +0x28b
reflect.Value.Set({0xa898020?, 0xc00172af20?, 0x416a7f?}, {0xa894480?, 0x0?, 0xc000e9c200?})
	reflect/value.go:2057 +0xe6
github.com/go-viper/mapstructure/v2.(*Decoder).decodeMapFromMap(0xc0017af140, {0x98c91f4, 0x7}, {0xa894480?, 0x0?, 0xc00172af20?}, {0xa898020?, 0xc00172af20?, 0x8?}, {0xa898020, ...})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:938 +0x2d3
github.com/go-viper/mapstructure/v2.(*Decoder).decodeMap(0xc0017af140, {0x98c91f4, 0x7}, {0xa894480, 0x0}, {0xa898020?, 0xc00172af20?, 0xc000e9c488?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:890 +0x2f0
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc0017af140, {0x98c91f4, 0x7}, {0x0, 0x0?}, {0xa898020?, 0xc00172af20?, 0x23?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:534 +0x[825](https://github.com/plengauer/Thoth/actions/runs/15320431855/job/43104184291#step:5:826)
github.com/go-viper/mapstructure/v2.(*Decoder).decodeStructFromMap(0xc0017af140, {0x0, 0x0}, {0xa894480?, 0xc00179b9e0?, 0xb6c2b20?}, {0xb6c2b20?, 0xc00172adc8?, 0xc000e9cef8?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1489 +0xb25
github.com/go-viper/mapstructure/v2.(*Decoder).decodeStruct(0xc0017af140, {0x0, 0x0}, {0xa894480?, 0xc00179b9e0?}, {0xb6c2b20?, 0xc00172adc8?, 0xa356f20?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1305 +0x55c
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0?}, {0xb6c2b20?, 0xc00172adc8?, 0xc001072d28?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:532 +0x94c
github.com/go-viper/mapstructure/v2.(*Decoder).decodePtr(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0}, {0xa356f20?, 0xc001072d28?, 0xc00176fdc0?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1131 +0x3f6
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0?}, {0xa356f20?, 0xc001072d28?, 0xc001072d28?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:536 +0x86e
github.com/go-viper/mapstructure/v2.(*Decoder).decodePtr(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0}, {0xc00176fdc0?, 0xc001072d28?, 0xc000870320?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1137 +0x49d
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0?}, {0xc00176fdc0?, 0xc001072d28?, 0xa894480?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:536 +0x86e
github.com/go-viper/mapstructure/v2.(*Decoder).decodeBasic(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0}, {0xa53bf00?, 0xc0017af060?, 0x7?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:582 +0x61f
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc0017af140, {0x0, 0x0}, {0xa894480, 0xc00179b9e0?}, {0xa53bf00?, 0xc0017af060?, 0xc0017b0360?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:520 +0x7d7
github.com/go-viper/mapstructure/v2.(*Decoder).Decode(0xc0017af140, {0xa894480, 0xc00179b9e0})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:434 +0xb4
go.opentelemetry.io/collector/confmap.decodeConfig(0xc0017aeef0, {0x9a845a0, 0xc0017af060}, 0x1, 0x0)
	go.opentelemetry.io/collector/confmap@v1.33.0/confmap.go:271 +0x339
go.opentelemetry.io/collector/confmap.(*Conf).Unmarshal(0xc0017aeef0, {0x9a845a0, 0xc0017af060}, {0x0, 0x0, 0xc0011874d9?})
	go.opentelemetry.io/collector/confmap@v1.33.0/confmap.go:96 +0xa5
go.opentelemetry.io/collector/otelcol/internal/configunmarshaler.(*Configs[...]).Unmarshal(0xd0bdd60, 0xc00179f[840](https://github.com/plengauer/Thoth/actions/runs/15320431855/job/43104184291#step:5:841))
	go.opentelemetry.io/collector/otelcol@v0.127.0/internal/configunmarshaler/configs.go:53 +0x271
go.opentelemetry.io/collector/confmap.decodeConfig.unmarshalerHookFunc.func7({0xa894480?, 0xc001776f00?, 0xc000e9db98?}, {0xae50c40?, 0xc00177c360?, 0xa894480?})
	go.opentelemetry.io/collector/confmap@v1.33.0/confmap.go:526 +0x264
go.opentelemetry.io/collector/confmap.decodeConfig.unmarshalerHookFunc.safeWrapDecodeHookFunc.func11({0xa894480?, 0xc001776f00?, 0xc00177c350?}, {0xae50c40?, 0xc00177c360?, 0xc001776f00?})
	go.opentelemetry.io/collector/confmap@v1.33.0/confmap.go:338 +0x88
go.opentelemetry.io/collector/confmap/internal/third_party/composehook.cachedDecodeHook.func3({0xa894480?, 0xc001776f00?, 0xd1d3be0?}, {0xae50c40?, 0xc00177c360?, 0x4198d4?})
	go.opentelemetry.io/collector/confmap@v1.33.0/internal/third_party/composehook/compose_hook.go:61 +0x2c
go.opentelemetry.io/collector/confmap/internal/third_party/composehook.ComposeDecodeHookFunc.func1({0xa894480?, 0xc001776f00?, 0xc0017052c0?}, {0xae50c40?, 0xc00177c360?, 0xc001776f00?})
	go.opentelemetry.io/collector/confmap@v1.33.0/internal/third_party/composehook/compose_hook.go:94 +0x107
github.com/go-viper/mapstructure/v2.cachedDecodeHook.func3({0xa894480?, 0xc001776f00?, 0x476c0c?}, {0xae50c40?, 0xc00177c360?, 0xa2731e0?})
	github.com/go-viper/mapstructure/v2@v2.2.1/decode_hooks.go:55 +0x2c
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc00177c460, {0x97fa2dd, 0x9}, {0xa894480, 0xc001776f00?}, {0xae50c40?, 0xc00177c360?, 0x16?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:505 +0x54b
github.com/go-viper/mapstructure/v2.(*Decoder).decodeStructFromMap(0xc00177c460, {0x0, 0x0}, {0xa894480?, 0xc001776de0?, 0xb5c49c0?}, {0xb5c49c0?, 0xc0010811e0?, 0xc000e9e750?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1489 +0xb25
github.com/go-viper/mapstructure/v2.(*Decoder).decodeStruct(0xc00177c460, {0x0, 0x0}, {0xa894480?, 0xc001776de0?}, {0xb5c49c0?, 0xc0010811e0?, 0x9a84ea0?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1305 +0x55c
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc00177c460, {0x0, 0x0}, {0xa894480, 0xc001776de0?}, {0xb5c49c0?, 0xc0010811e0?, 0xc001072638?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:532 +0x94c
github.com/go-viper/mapstructure/v2.(*Decoder).decodePtr(0xc00177c460, {0x0, 0x0}, {0xa894480, 0xc001776de0}, {0x9a84ea0?, 0xc001072638?, 0xa?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:1131 +0x3f6
github.com/go-viper/mapstructure/v2.(*Decoder).decode(0xc00177c460, {0x0, 0x0}, {0xa894480, 0xc001776de0?}, {0x9a84ea0?, 0xc001072638?, 0x0?})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:536 +0x86e
github.com/go-viper/mapstructure/v2.(*Decoder).Decode(0xc00177c460, {0xa894480, 0xc001776de0})
	github.com/go-viper/mapstructure/v2@v2.2.1/mapstructure.go:434 +0xb4
go.opentelemetry.io/collector/confmap.decodeConfig(0xc00176d450, {0x9bb8a40, 0xc001072638}, 0x1, 0x0)
	go.opentelemetry.io/collector/confmap@v1.33.0/confmap.go:271 +0x339
go.opentelemetry.io/collector/confmap.(*Conf).Unmarshal(0xc00176d450, {0x9bb8a40, 0xc001072638}, {0x0, 0x0, 0xc001776000?})
	go.opentelemetry.io/collector/confmap@v1.33.0/confmap.go:96 +0xa5
go.opentelemetry.io/collector/otelcol.unmarshal(0xc00176d450, {0xc00170d530, 0xc00170d920, 0xc00170d770, 0xc00170d3b0, 0xc00170da10, 0xc00170d560, 0xc00170d950, 0xc00170d7a0, 0xc00170d3e0, ...})
	go.opentelemetry.io/collector/otelcol@v0.127.0/unmarshaler.go:46 +0x2f9
go.opentelemetry.io/collector/otelcol.(*ConfigProvider).Get(0xc001669ea0?, {0xd0ebc00?, 0x14705440?}, {0xc00170d530, 0xc00170d920, 0xc00170d770, 0xc00170d3b0, 0xc00170da10, 0xc00170d560, 0xc00170d950, ...})
	go.opentelemetry.io/collector/otelcol@v0.127.0/configprovider.go:62 +0xb8
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents(0xc001080580, {0xd0ebc00, 0x14705440})
	go.opentelemetry.io/collector/otelcol@v0.127.0/collector.go:180 +0x150
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0xc001080580, {0xd0ebc00, 0x14705440})
	go.opentelemetry.io/collector/otelcol@v0.127.0/collector.go:312 +0x55
go.opentelemetry.io/collector/otelcol.NewCommand.func1(0xc00172e908, {0xbd365c1?, 0x7?, 0xbd2a43a?})
	go.opentelemetry.io/collector/otelcol@v0.127.0/command.go:39 +0x94
github.com/spf13/cobra.(*Command).execute(0xc00172e908, {0xc000121ae0, 0x2, 0x2})
	github.com/spf13/cobra@v1.9.1/command.go:1015 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00172e908)
	github.com/spf13/cobra@v1.9.1/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(0xc1b6998?)
	github.com/spf13/cobra@v1.9.1/command.go:1071 +0x13
main.runInteractive({0xc1b6998, {{0xbd79765, 0xf}, {0xbe4b467, 0x1f}, {0xbd36408, 0x7}, {}}, 0x0, {{{0x0, ...}, ...}, ...}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:70 +0x4a
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:10
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:63 +0x598

Additional context

No response

[songy23]

cc @mx-psi could it be #12996 (the only change in confmap in v0.127)

[mx-psi]

I can reproduce, the following more minimal configuration reproduces the issue:

receivers:
  nop:
exporters:
  otlphttp:
    endpoint: "http://localhost:8080"
    headers:

service:
  pipelines:
    traces:
      receivers: [nop]
      exporters: [otlphttp]

while this one (without headers:) does not:

receivers:
  nop:
exporters:
  otlphttp:
    endpoint: "http://localhost:8080"

service:
  pipelines:
    traces:
      receivers: [nop]
      exporters: [otlphttp]

If that's not possible, a workaround would be to set headers: {} (headers: null does trigger the panic)

[mx-psi]

@plengauer Would your setup allow you to either remove the empty headers option or set headers to {}?

[atoulme]

I reproduce with a small unit test:

type structWithConfigOpaqueMap struct {
	Foo     string                         `mapstructure:"foo"`
	Headers map[string]configopaque.String `mapstructure:"headers"`
}

func TestNilConfigOpaque(t *testing.T) {
	s := structWithConfigOpaqueMap{
		Headers: map[string]configopaque.String{},
	}
	var data map[string]any
	require.NoError(t, yaml.Unmarshal([]byte("foo: foo\nheaders:"), &data), "unable to parse yaml")
	c := NewFromStringMap(data)
	require.NoError(t, c.Unmarshal(&s))
	assert.Equal(t, s.Foo, "foo")
	assert.Nil(t, s.Headers)
}

[atoulme]

Reverting #12996 locally makes the test pass.

[mx-psi]

Interesting, this reproduces it:

type structWithConfigOpaqueMap struct {
	Foo     string            `mapstructure:"foo"`
	Headers map[string]string `mapstructure:"headers"`
}

func TestNilConfigOpaque(t *testing.T) {
	s := structWithConfigOpaqueMap{
		Headers: map[string]string{},
	}
	var data map[string]any
	require.NoError(t, yaml.Unmarshal([]byte("foo: foo\nheaders:"), &data), "unable to parse yaml")
	c := NewFromStringMap(data)
	require.NoError(t, c.Unmarshal(&s))
	assert.Equal(t, s.Foo, "foo")
	assert.Nil(t, s.Headers)
}

but this one does not:

type structWithConfigOpaqueMap struct {
	Foo     string            `mapstructure:"foo"`
	Headers map[string]string `mapstructure:"headers"`
}

func TestNilConfigOpaque(t *testing.T) {
	s := structWithConfigOpaqueMap{
		Headers: nil, // <-- nil works
	}
	var data map[string]any
	require.NoError(t, yaml.Unmarshal([]byte("foo: foo\nheaders:"), &data), "unable to parse yaml")
	c := NewFromStringMap(data)
	require.NoError(t, c.Unmarshal(&s))
	assert.Equal(t, s.Foo, "foo")
	assert.Nil(t, s.Headers)
}

[atoulme]

That's right. The default value must be set, otherwise the problem doesn't happen.

[mx-psi]

The simplest unit test I can come up with is the following:

type structWithStringMap struct {
	Headers map[string]string `mapstructure:"headers"`
}

func TestNilConfigOpaque(t *testing.T) {
	s := structWithStringMap{
		Headers: map[string]string{},
	}
	c := NewFromStringMap(map[string]any{
		"headers": nil,
	})
	require.NoError(t, c.Unmarshal(&s))
	assert.Nil(t, s.Headers)
}

[atoulme]

Here is a fix?

func zeroSliceHookFunc() mapstructure.DecodeHookFuncValue {
	return safeWrapDecodeHookFunc(func(from reflect.Value, to reflect.Value) (any, error) {
		if to.CanSet() && to.Kind() == reflect.Slice && from.Kind() == reflect.Slice {
			if !from.IsNil() {
				// input slice is not nil, set the output slice to a new slice of the same type.
				to.Set(reflect.MakeSlice(to.Type(), from.Len(), from.Cap()))
			}
		}
		if to.CanSet() && to.Kind() == reflect.Map && from.Kind() == reflect.Map {
			if !to.IsNil() && to.Len() == 0 && from.IsNil() {
				// input slice is nil, set the output slice to nil.
				to.SetZero()
			}
		}

		return from.Interface(), nil
	})

[atoulme]

this adds

if to.CanSet() && to.Kind() == reflect.Map && from.Kind() == reflect.Map {
			if !to.IsNil() && to.Len() == 0 && from.IsNil() {
				// input slice is nil, set the output slice to nil.
				to.SetZero()
			}
		}

where if !to.IsNil() && to.Len() == 0 && from.IsNil() means: the target map is not nil, but empty, and the from map is nil, then set the target map as nil.