Document how consumers filtering logs on severity should treat unset or unspecified (0) severity

[lmolkova]

According to proto definition, default severity value is 0 -

// UNSPECIFIED is the default SeverityNumber, it MUST NOT be used.
SEVERITY_NUMBER_UNSPECIFIED = 0;

The property is optional for the producers - they can leave it unset, but on the proto level it translates to 0.
Consumers should expect to always receive this value.

This introduces the following questions in the context of log filtering. Assuming user has set severity threshold to N:

1. when SDK log filtering processor gets a log record without severity, should it drop it?
2. when collector log filtering processor gets a log record with severity 0, should it drop it?
3. when backend receives a log record with severity 0, should it make it comparable to other logs record severities
4. when user writes a query, how do they filter logs based on severity numbers?

In all cases the question boils down to what's the comparison algorithm for severity

1. This algorithm has to be the same for all cases mentioned above. Treating unset (in SDK processor) severity differently than 0 (in collector/backend/query) will lead to compatibility problems and errors.
2. Relying on int comparison seems to be the most natural option
   • it covers full range of possible severity number values (including negatives and 0)
   • it's trivial to implement and is natural for anyone seeing int or protobuf enum to assume they can just rely on severity < threshold and default sorting algorithms and compare-to implementations
   • special-casing severities <= 0 means everyone needs custom compare-to implementation - severity > 0 && severity < threshold
3. Telemetry producers should do their best effort to provide reasonable severity number. When they don't set it, they're effectively assuming that general-purpose consumers that apply log filtering may drop corresponding logs.

Proposal

1. we should remove limitation for comparison

   opentelemetry-specification/specification/logs/data-model.md

   Lines 420 to 423 in bdcf53d

   	In the contexts where severity participates in less-than / greater-than
   	comparisons `SeverityNumber` field should be used. `SeverityNumber` can be
   	compared to another `SeverityNumber` or to numbers in the 1..24 range (or to the
   	corresponding short names).

   - compared to another `SeverityNumber` or to numbers in the 1..24 range (or to the
   - corresponding short names).
   + compared to another `SeverityNumber` or to an integer.

   I don't even think it's breaking - there is no normative language here and no mention that values outside of the range can't be
   compared to each other. It'd be very surprising to assume that 42 is less than 41 (or that 0 is the largest)

2. we should encourage producers to set severity. Semconv would recommend setting severity on all events, but the value is usually not fixed (e.g. exception may have severity from debug to fatal).

3. Update

   opentelemetry-specification/specification/logs/data-model.md

   Lines 331 to 335 in bdcf53d

   	Source formats that do not define a concept of severity or log level MAY omit
   	`SeverityNumber` and `SeverityText` fields. Backend and UI may represent log
   	records with missing severity information distinctly or may interpret log
   	records with missing `SeverityNumber` and `SeverityText` fields as if the
   	`SeverityNumber` was set equal to INFO (numeric value of 9).

   - Source formats that do not define a concept of severity or log level MAY omit
   - `SeverityNumber` and `SeverityText` fields. Backend and UI may represent log
   - records with missing severity information distinctly or may interpret log
   - records with missing `SeverityNumber` and `SeverityText` fields as if the
   - `SeverityNumber` was set equal to INFO (numeric value of 9).
   + Log records without severity number MAY be dropped by log filters depending on the user-configured threshold. 
   + It's RECOMMENDED for all log producers to provide a severity number even if source format does not define a 
   + concept of severity or log level.

   Current guidance is not strict and is questionable as a common strategy to treat 0 as 9, and anyway it's totally on the Backend and UI side to treat 0 as anything they want.

4. We should allow consumers and any customization layers to special-case 0 or any other severity number and write custom log filtering and comparison logic. This is also a back-compat path forward to anyone who special-cased unset severity as log-never or log-always. Effectively it leaves behavior in case of 0 undefined.

[pellared]

Escalating as Go SIG maintainer as it is related to (or even blocking):

• [DO NOT MERGE] minsev: Fix to filter out only in SeverityTrace1..SeverityFatal4 severity range opentelemetry-go-contrib#7424