[connector/routing] Not included in contrib distribution

[yanamg7]

Component(s)

connector/routing

What happened?

Using artifactory.f5net.com/dockerhub-remote/otel/opentelemetry-collector-contrib:0.84.0 I trying to add connector routing

Steps to Reproduce

I added to config.yaml this part:
connector:
routing:
default_pipelines: [logs/all]
error_mode: ignore
table:
- statement: route() where resource.attributes["Severity"] == "err"
pipelines: [logs/err]
- statement: route() where resource.attributes["Severity"] == "info"
pipelines: [logs/info]

Trying to run

Expected Result

Run successfully, no errors, correct output

Actual Result

error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])

Collector version

0.84.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ /opt/parser/*.log ]
    start_at: beginning
    operators:
      - type: add
        field: attributes.sev
        value: unspecified
      - type: json_parser
      - type: severity_parser
        parse_from: attributes.sev
        mapping:
          fatal: emergency
          info4: notice
      - type: move
        id: move5
        from: attributes.sev
        to: resource["Severity"]
      

exporters:
  file/file1:
    path: /var/log/collector/logging1/otlp.json
  file/file2:
    path: /var/log/collector/logging2/otlp.json
  file/file3:
    path: /var/log/collector/logging3/otlp.json
  logging:
    loglevel: debug


connector:
  routing:
      default_pipelines: [logs/all]
      error_mode: ignore
      table:
        - statement: route() where resource.attributes["Severity"] == "err"
          pipelines: [logs/err]
        - statement: route() where resource.attributes["Severity"] == "info"
          pipelines: [logs/info]

service:
  pipelines:
    logs/in:
      receivers: [filelog]
      exporters: [routing, logging]
    logs/all:
      receivers: [routing]
      exporters: [file/file3, logging]
    logs/info:
      receivers: [routing]
      processors: []
      exporters: [file/file1, logging]
    logs/err:
      receivers: [routing]
      processors: []
      exporters: [file/file2, logging]

Log output

otel-contrib  | Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-contrib  | 
otel-contrib  | * error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
otel-contrib  | 2023/09/07 07:53:12 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-contrib  | 
otel-contrib  | * error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
otel-contrib exited with code 1

Additional context

No response

[github-actions]

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mwear]

@yanamg7 can you make sure that connectors ends in an "s" (rename connector -> connectors) in your yaml?

connectors:
  routing:
      default_pipelines: [logs/all]
      error_mode: ignore
      table:
        - statement: route() where resource.attributes["Severity"] == "err"
          pipelines: [logs/err]
        - statement: route() where resource.attributes["Severity"] == "info"
          pipelines: [logs/info]

[mwear]

Actually scrap that idea @yanamg7. I was able to reproduce your error with similar yaml. I will look into this further and let you know what I find.

[mwear]

@yanamg7 after taking a closer look, the issue is that the routing connector is still in development and components with development status are not included in the collector contrib public images.

opentelemetry-collector-contrib/connector/routingconnector/metadata.yaml

Line 6 in 9424ae3

development: [traces_to_traces, metrics_to_metrics, logs_to_logs]

If you want to use the routing connector today, you will have to build your own image. You could use the collector builder to do this.

The reason the status of this component is development is that there is still some major feature work that needs to take place. Most notably, context based routing. If you are interested using the routing connector in its bleeding edge state, and would like it to be part of the public contrib images, let me know and we can consider changing the status to alpha.

[yanamg7]

Hi @mwer
Thank you for the quick response,
We would much appreciate it if you could include the routing processor in the public image/build,
It would be beneficial for us.
We are currently investigating routing options for our OTEL, and we are developing some PoC for several routing scenarios.
Would you happen to have a recommended method to compare different setups of OTEL Collectors?
We are considering several options for routing our Resources (Logs and Metrics) based on Resources attributes.
We want a method to measure each setup based on resource consumption among other criteria
We would appreciate your advice

[jpkrohling]

While I'd love the connector to be ready, the routing processor is still there. Is there a specific reason you can't use it?

[mwear]

@yanamg7, as @jpkrohling points out, the routing processor exists and more feature complete. The goal is for the routing connector to eventually replace it, but there is more work that needs to be done for it to have parity. Let us know if the processor will work for you for now or if you would still prefer to use the connector.

[nadavleva]

Hi @mwear and @jpkrohling, I am working with @yanamg7, on the same project.

Thank you for your quick response.

We are checking several routing options for our OTEL Collector based on resources and attributes. We are currently working with Metrics and Logs resource types.

In our current implementation, we allow the dynamic creation of Otlp Exporters using API and adding these exporters to the pipelines based on the API input.

The routing connector seems to be better suited to our needs, since it routes to Pipelines and not to Exporters.

In case we will use the Routing Processor then, the exporter creation will have to be embedded in the routing logic.

In addition, the routing processer is going to be deprecated according to our understanding of this #19738

I have some questions:
1. Regarding the Routing Connector, what is the ETA for its release?
2. Is the Routing processor deprecation decision final or both the routing connector and processor will be available?
3. In case we want to contribute to the development of the Otel Collector, especially the routing connector, what is the procedure and policies to become an active contributor?