Skip to content

Fix Couchbase flaky tests #6892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 24, 2025
Merged

Fix Couchbase flaky tests #6892

merged 3 commits into from
Apr 24, 2025

Conversation

NachoEchevarria
Copy link
Contributor

@NachoEchevarria NachoEchevarria commented Apr 24, 2025 •
edited
Loading

Summary of changes

The CouchBase and CouchBase3 tests are failing rarely in the CI due to authentication errors. All of them seem to be due to Authorization issues, not related to our instrumentation. That's why it has been decided to skip these results when the flakiness occurs.

In the case of the CouchBase tests, we are getting these stacktraces depending on the package version:

Unhandled exception. System.AggregateException: Could not bootstrap - check inner exceptions for details. (One or more errors occurred. (Authentication failed for bucket 'default')) (Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'Couchbase.Configuration.Server.Serialization.Settings' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.
Path 'settings', line 1, position 88.)

or

Unhandled exception. Couchbase.Configuration.Server.Serialization.BootstrapException: Could not bootstrap - check inner exceptions for details. (Authentication failed for bucket 'default') (Could not bootstrap with CCCP. (Authentication failed for bucket 'default')) (Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'Couchbase.Configuration.Server.Serialization.Settings' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
 To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.
Path 'settings', line 1, position 88.)

Couchbase.Configuration.Server.Serialization.BootstrapException inherits from System.AggregateException.

In the case of Couuchbase3, we are getting this exception:

Unhandled exception. Couchbase.AuthenticationFailureException: Cannot authenticate the user. Reason: AuthenticationError
at Couchbase.Core.IO.Authentication.SaslMechanismBase.SendAsync[T](IOperation`1 op, IConnection connection, CancellationToken cancellationToken)
at Couchbase.Core.IO.Authentication.SaslMechanismBase.SaslStep(IConnection connection, String message, IRequestSpan span, CancellationToken token)
at Couchbase.Core.IO.Authentication.ScramShaMechanism.AuthenticateAsync(IConnection connection, CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.Couchbase.Core.IO.Connections.IConnectionInitializer.InitializeConnectionAsync(IConnection connection, CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.ConnectionPoolBase.CreateConnectionAsync(CancellationToken cancellationToken)

By returning a 13 return code, we communicate the test framework that the text can be skipped. This code is processed in \tracer\test\Datadog.Trace.TestHelpers.AutoInstrumentation\ErrorHelpers.cs

Reason for change

Implementation details

Test coverage

Other details

@github-actions github-actions bot added the area:tests unit tests, integration tests label Apr 24, 2025
@NachoEchevarria NachoEchevarria changed the title Add catch for exceptions and return 13. Fix Couchbase flaky tests Apr 24, 2025
@pr-commenter
Copy link

pr-commenter bot commented Apr 24, 2025

Benchmarks

Benchmarks Report for tracer 🐌

Benchmarks for #6892 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.167
  • 1 benchmarks have fewer allocations
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6892

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net6.0 5.51 KB 5.55 KB 40 B 0.73%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.7μs 57.9ns 338ns 0 0 0 5.51 KB
master StartStopWithChild netcoreapp3.1 13.5μs 67.9ns 311ns 0 0 0 5.72 KB
master StartStopWithChild net472 22.6μs 121ns 660ns 0.965 0.214 0 6.08 KB
#6892 StartStopWithChild net6.0 10.9μs 53ns 218ns 0 0 0 5.55 KB
#6892 StartStopWithChild netcoreapp3.1 13.6μs 68.2ns 297ns 0 0 0 5.75 KB
#6892 StartStopWithChild net472 22.4μs 127ns 850ns 0.97 0.323 0.108 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6892

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 3.35 KB 3.38 KB 28 B 0.84%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 942μs 174ns 602ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 440ns 1.71μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.21ms 151ns 544ns 0 0 0 3.35 KB
#6892 WriteAndFlushEnrichedTraces net6.0 936μs 309ns 1.12μs 0 0 0 2.71 KB
#6892 WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 457ns 1.77μs 0 0 0 2.7 KB
#6892 WriteAndFlushEnrichedTraces net472 1.25ms 86.2ns 311ns 0 0 0 3.38 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 62.3μs 39.4ns 142ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 70.3μs 31.9ns 119ns 0 0 0 17.43 KB
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#6892 SendRequest net6.0 61.5μs 38.6ns 144ns 0 0 0 14.53 KB
#6892 SendRequest netcoreapp3.1 70.5μs 200ns 722ns 0 0 0 17.42 KB
#6892 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 Fewer allocations 🎉

Faster 🎉 in #6892

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.167 746,840.80 640,153.57

Fewer allocations 🎉 in #6892

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 42.29 KB 41.89 KB -405 B -0.96%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 703μs 1.1μs 4.24μs 0 0 0 41.9 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 715μs 4.96μs 49.3μs 0 0 0 42.29 KB
master WriteAndFlushEnrichedTraces net472 914μs 2.81μs 10.9μs 8.33 0 0 56.42 KB
#6892 WriteAndFlushEnrichedTraces net6.0 631μs 1.95μs 7.01μs 0 0 0 41.73 KB
#6892 WriteAndFlushEnrichedTraces netcoreapp3.1 641μs 1.32μs 6.61μs 0 0 0 41.89 KB
#6892 WriteAndFlushEnrichedTraces net472 945μs 3.21μs 12.4μs 4.46 0 0 56.17 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.88μs 9.66ns 44.3ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.56μs 12.2ns 53.4ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.77μs 2.89ns 10.8ns 0.153 0.0139 0 995 B
#6892 ExecuteNonQuery net6.0 2.04μs 6.9ns 26.7ns 0 0 0 1.03 KB
#6892 ExecuteNonQuery netcoreapp3.1 2.48μs 8.57ns 32ns 0 0 0 1.02 KB
#6892 ExecuteNonQuery net472 2.81μs 4.93ns 19.1ns 0.154 0.014 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.79μs 0.327ns 1.13ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.28μs 9.62ns 37.3ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.49μs 1.52ns 5.48ns 0.157 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.82μs 1.69ns 6.53ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.42μs 12ns 53.6ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.58μs 0.971ns 3.76ns 0.161 0 0 1.11 KB
#6892 CallElasticsearch net6.0 1.87μs 3.02ns 11.7ns 0 0 0 1.04 KB
#6892 CallElasticsearch netcoreapp3.1 2.25μs 6.79ns 24.5ns 0 0 0 1.04 KB
#6892 CallElasticsearch net472 3.54μs 1.22ns 4.56ns 0.159 0 0 1.05 KB
#6892 CallElasticsearchAsync net6.0 1.84μs 8.99ns 40.2ns 0 0 0 1.02 KB
#6892 CallElasticsearchAsync netcoreapp3.1 2.39μs 12ns 50.8ns 0 0 0 1.09 KB
#6892 CallElasticsearchAsync net472 3.67μs 4.24ns 16.4ns 0.164 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.82μs 6.9ns 25.8ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.25μs 1.49ns 5.37ns 0 0 0 960 B
master ExecuteAsync net472 2.53μs 0.761ns 2.95ns 0.14 0 0 923 B
#6892 ExecuteAsync net6.0 1.82μs 8.66ns 35.7ns 0 0 0 960 B
#6892 ExecuteAsync netcoreapp3.1 2.3μs 10.8ns 43.2ns 0 0 0 960 B
#6892 ExecuteAsync net472 2.62μs 1.14ns 4.43ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.82μs 10.5ns 40.5ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.31μs 23.5ns 87.8ns 0 0 0 2.9 KB
master SendAsync net472 12.3μs 8.81ns 34.1ns 0.496 0 0 3.19 KB
#6892 SendAsync net6.0 7.15μs 6.38ns 22.1ns 0 0 0 2.37 KB
#6892 SendAsync netcoreapp3.1 8.52μs 15.4ns 59.8ns 0 0 0 2.9 KB
#6892 SendAsync net472 12.4μs 6.88ns 25.7ns 0.495 0 0 3.19 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.63μs 0.927ns 3.59ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.47μs 4.15ns 16.1ns 0 0 0 1.76 KB
master EnrichedLog net472 3.88μs 6.02ns 23.3ns 0.252 0 0 1.69 KB
#6892 EnrichedLog net6.0 2.58μs 13.7ns 68.4ns 0 0 0 1.76 KB
#6892 EnrichedLog netcoreapp3.1 3.56μs 3.8ns 14.7ns 0 0 0 1.76 KB
#6892 EnrichedLog net472 3.93μs 4.21ns 16.3ns 0.256 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 124μs 73.9ns 277ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 129μs 328ns 1.27μs 0 0 0 4.37 KB
master EnrichedLog net472 170μs 132ns 493ns 0 0 0 4.57 KB
#6892 EnrichedLog net6.0 125μs 119ns 462ns 0 0 0 4.37 KB
#6892 EnrichedLog netcoreapp3.1 129μs 375ns 1.45μs 0 0 0 4.37 KB
#6892 EnrichedLog net472 170μs 195ns 729ns 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.15μs 6.42ns 24ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.57μs 18.6ns 72.2ns 0 0 0 2.32 KB
master EnrichedLog net472 7.34μs 5.98ns 23.2ns 0.329 0 0 2.14 KB
#6892 EnrichedLog net6.0 5.09μs 10.4ns 40.3ns 0 0 0 2.32 KB
#6892 EnrichedLog netcoreapp3.1 6.48μs 17.8ns 68.8ns 0 0 0 2.32 KB
#6892 EnrichedLog net472 7.25μs 6.09ns 23.6ns 0.328 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.02μs 10.1ns 42.7ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.83μs 13.4ns 53.5ns 0 0 0 1.21 KB
master SendReceive net472 3.08μs 6.37ns 24.7ns 0.183 0 0 1.21 KB
#6892 SendReceive net6.0 1.99μs 9.23ns 35.7ns 0 0 0 1.21 KB
#6892 SendReceive netcoreapp3.1 2.66μs 11.8ns 45.7ns 0 0 0 1.21 KB
#6892 SendReceive net472 3.07μs 2.78ns 10.4ns 0.183 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.09μs 7.9ns 29.6ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.63μs 13.3ns 49.8ns 0 0 0 1.69 KB
master EnrichedLog net472 6.59μs 6.86ns 26.6ns 0.328 0 0 2.08 KB
#6892 EnrichedLog net6.0 4.24μs 2.45ns 9.48ns 0 0 0 1.64 KB
#6892 EnrichedLog netcoreapp3.1 5.82μs 7.96ns 30.8ns 0 0 0 1.69 KB
#6892 EnrichedLog net472 6.63μs 6.18ns 23.1ns 0.331 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 744ns 3.75ns 18ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 932ns 4.72ns 21.6ns 0 0 0 584 B
master StartFinishSpan net472 919ns 0.192ns 0.744ns 0.0922 0 0 586 B
master StartFinishScope net6.0 938ns 4.92ns 24.1ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.16μs 5.96ns 29.8ns 0 0 0 705 B
master StartFinishScope net472 1.11μs 0.0574ns 0.207ns 0.105 0 0 666 B
#6892 StartFinishSpan net6.0 756ns 0.38ns 1.47ns 0 0 0 584 B
#6892 StartFinishSpan netcoreapp3.1 950ns 5.21ns 29ns 0 0 0 584 B
#6892 StartFinishSpan net472 918ns 0.251ns 0.94ns 0.0917 0 0 586 B
#6892 StartFinishScope net6.0 911ns 0.477ns 1.85ns 0 0 0 704 B
#6892 StartFinishScope netcoreapp3.1 1.15μs 5.26ns 21ns 0 0 0 704 B
#6892 StartFinishScope net472 1.11μs 0.182ns 0.706ns 0.105 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.02μs 5.56ns 34.3ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.43μs 0.863ns 3.34ns 0 0 0 704 B
master RunOnMethodBegin net472 1.35μs 0.553ns 2.14ns 0.1 0 0 666 B
#6892 RunOnMethodBegin net6.0 1.01μs 5.2ns 24.4ns 0 0 0 704 B
#6892 RunOnMethodBegin netcoreapp3.1 1.4μs 5.57ns 21.6ns 0 0 0 705 B
#6892 RunOnMethodBegin net472 1.35μs 2.19ns 8.46ns 0.0999 0 0 666 B

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Apr 24, 2025 •
edited
Loading

Datadog Report

Branch report: nacho/CouchBaseExceptions
Commit report: 2c85f58
Test service: dd-trace-dotnet

✅ 0 Failed, 251804 Passed, 2291 Skipped, 18h 4m 40.8s Total Time

@andrewlock
Copy link
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6892 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.123
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6892

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net6.0 5.57 KB 5.65 KB 84 B 1.51%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.73 KB 5.76 KB 29 B 0.51%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.74μs 90.1ns 896ns 0 0 0 5.57 KB
master StartStopWithChild netcoreapp3.1 10.1μs 148ns 1.48μs 0 0 0 5.73 KB
master StartStopWithChild net472 15.3μs 87.8ns 697ns 1.04 0.297 0.0741 6.13 KB
#6892 StartStopWithChild net6.0 9.55μs 267ns 2.66μs 0 0 0 5.65 KB
#6892 StartStopWithChild netcoreapp3.1 9.88μs 110ns 1.08μs 0.0476 0 0 5.76 KB
#6892 StartStopWithChild net472 14.4μs 66.9ns 276ns 0.929 0.155 0 6.11 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 513μs 5.69μs 50.6μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 664μs 854ns 3.19μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 841μs 2.7μs 10.1μs 0 0 0 3.31 KB
#6892 WriteAndFlushEnrichedTraces net6.0 526μs 1.85μs 6.67μs 0 0 0 2.71 KB
#6892 WriteAndFlushEnrichedTraces netcoreapp3.1 667μs 1.18μs 4.27μs 0 0 0 2.7 KB
#6892 WriteAndFlushEnrichedTraces net472 867μs 340ns 1.27μs 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 127μs 626ns 2.66μs 0 0 0 14.48 KB
master SendRequest netcoreapp3.1 141μs 821ns 7.15μs 0 0 0 17.28 KB
master SendRequest net472 0.00158ns 0.000518ns 0.00201ns 0 0 0 0 b
#6892 SendRequest net6.0 129μs 735ns 5.2μs 0 0 0 14.54 KB
#6892 SendRequest netcoreapp3.1 148μs 240ns 898ns 0 0 0 17.28 KB
#6892 SendRequest net472 0.00191ns 0.00063ns 0.00236ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 653μs 7.16μs 70.9μs 0 0 0 41.81 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 745μs 8.26μs 80.5μs 0 0 0 41.86 KB
master WriteAndFlushEnrichedTraces net472 871μs 4.95μs 46.7μs 7.35 3.68 0 53.37 KB
#6892 WriteAndFlushEnrichedTraces net6.0 615μs 6.35μs 61.5μs 0 0 0 41.69 KB
#6892 WriteAndFlushEnrichedTraces netcoreapp3.1 724μs 6.75μs 66.1μs 0 0 0 41.74 KB
#6892 WriteAndFlushEnrichedTraces net472 921μs 6.21μs 61.8μs 7.81 0 0 53.44 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.3μs 3.03ns 11.3ns 0.0129 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 1.78μs 3.2ns 11.1ns 0.00901 0 0 1.02 KB
master ExecuteNonQuery net472 2.15μs 1.56ns 5.83ns 0.153 0.0109 0 995 B
#6892 ExecuteNonQuery net6.0 1.33μs 1.14ns 3.96ns 0.0133 0 0 1.03 KB
#6892 ExecuteNonQuery netcoreapp3.1 1.76μs 2.1ns 8.12ns 0.00879 0 0 1.02 KB
#6892 ExecuteNonQuery net472 2.09μs 2.58ns 9.65ns 0.156 0.0104 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.32μs 1.17ns 4.53ns 0.0133 0 0 984 B
master CallElasticsearch netcoreapp3.1 1.6μs 1.34ns 4.83ns 0.00809 0 0 984 B
master CallElasticsearch net472 2.58μs 1.38ns 4.98ns 0.154 0 0 1 KB
master CallElasticsearchAsync net6.0 1.28μs 1.58ns 6.13ns 0.0126 0 0 960 B
master CallElasticsearchAsync netcoreapp3.1 1.73μs 1.34ns 5.17ns 0.00864 0 0 1.03 KB
master CallElasticsearchAsync net472 2.6μs 1.17ns 4.53ns 0.156 0 0 1.06 KB
#6892 CallElasticsearch net6.0 1.22μs 1.21ns 4.69ns 0.0123 0 0 984 B
#6892 CallElasticsearch netcoreapp3.1 1.61μs 1.82ns 7.04ns 0.00805 0 0 984 B
#6892 CallElasticsearch net472 2.53μs 0.773ns 2.89ns 0.152 0 0 1 KB
#6892 CallElasticsearchAsync net6.0 1.27μs 1.16ns 4.49ns 0.0128 0 0 960 B
#6892 CallElasticsearchAsync netcoreapp3.1 1.57μs 1.24ns 4.66ns 0.00795 0 0 1.03 KB
#6892 CallElasticsearchAsync net472 2.7μs 2.16ns 8.09ns 0.162 0 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.35μs 0.862ns 3.23ns 0.0135 0 0 960 B
master ExecuteAsync netcoreapp3.1 1.62μs 1.7ns 6.12ns 0.00819 0 0 960 B
master ExecuteAsync net472 1.81μs 0.999ns 3.74ns 0.145 0 0 923 B
#6892 ExecuteAsync net6.0 1.36μs 1.03ns 3.99ns 0.0136 0 0 960 B
#6892 ExecuteAsync netcoreapp3.1 1.66μs 1.69ns 6.1ns 0.00823 0 0 960 B
#6892 ExecuteAsync net472 1.86μs 1.37ns 5.32ns 0.138 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.4μs 1.76ns 6.09ns 0.0221 0 0 2.32 KB
master SendAsync netcoreapp3.1 5.29μs 9.69ns 37.5ns 0.0263 0 0 2.86 KB
master SendAsync net472 7.45μs 5.2ns 19.5ns 0.49 0 0 3.13 KB
#6892 SendAsync net6.0 4.38μs 3.43ns 12.8ns 0.022 0 0 2.32 KB
#6892 SendAsync netcoreapp3.1 5.37μs 4.53ns 17ns 0.0267 0 0 2.86 KB
#6892 SendAsync net472 7.49μs 3.91ns 15.2ns 0.487 0 0 3.13 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.68μs 1.82ns 7.07ns 0.0168 0 0 1.71 KB
master EnrichedLog netcoreapp3.1 2.34μs 1.83ns 7.09ns 0.0118 0 0 1.71 KB
master EnrichedLog net472 2.65μs 1.81ns 7.02ns 0.25 0 0 1.64 KB
#6892 EnrichedLog net6.0 1.51μs 0.889ns 3.2ns 0.0226 0 0 1.71 KB
#6892 EnrichedLog netcoreapp3.1 2.25μs 2.84ns 10.6ns 0.0224 0 0 1.71 KB
#6892 EnrichedLog net472 2.61μs 1.66ns 6.43ns 0.259 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 295ns 1.14μs 0 0 0 4.32 KB
master EnrichedLog netcoreapp3.1 115μs 424ns 1.64μs 0 0 0 4.32 KB
master EnrichedLog net472 149μs 226ns 876ns 0 0 0 4.51 KB
#6892 EnrichedLog net6.0 114μs 535ns 2.07μs 0 0 0 4.32 KB
#6892 EnrichedLog netcoreapp3.1 115μs 214ns 830ns 0 0 0 4.32 KB
#6892 EnrichedLog net472 151μs 205ns 768ns 0 0 0 4.51 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.26μs 2.4ns 9.29ns 0.0331 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 4.27μs 5.13ns 19.9ns 0.0214 0 0 2.26 KB
master EnrichedLog net472 4.89μs 3.09ns 12ns 0.318 0 0 2.09 KB
#6892 EnrichedLog net6.0 3.12μs 3.61ns 13.5ns 0.0306 0 0 2.26 KB
#6892 EnrichedLog netcoreapp3.1 4.35μs 3.21ns 11.6ns 0.0217 0 0 2.26 KB
#6892 EnrichedLog net472 4.89μs 3.34ns 12.9ns 0.319 0 0 2.09 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.37μs 1.14ns 4.41ns 0.0138 0 0 1.15 KB
master SendReceive netcoreapp3.1 1.8μs 1.13ns 4.22ns 0.00907 0 0 1.15 KB
master SendReceive net472 2.16μs 1.57ns 6.06ns 0.179 0 0 1.16 KB
#6892 SendReceive net6.0 1.38μs 0.868ns 3.25ns 0.0137 0 0 1.15 KB
#6892 SendReceive netcoreapp3.1 1.81μs 1.67ns 6.25ns 0.00898 0 0 1.15 KB
#6892 SendReceive net472 2.13μs 1.3ns 5.04ns 0.18 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.83μs 1.73ns 6.69ns 0.0142 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 3.9μs 2.18ns 7.85ns 0.0194 0 0 1.69 KB
master EnrichedLog net472 4.46μs 2.75ns 10.6ns 0.313 0 0 2.08 KB
#6892 EnrichedLog net6.0 2.81μs 2.26ns 8.77ns 0.0141 0 0 1.64 KB
#6892 EnrichedLog netcoreapp3.1 4.04μs 3.93ns 14.2ns 0.0203 0 0 1.69 KB
#6892 EnrichedLog net472 4.52μs 4.55ns 17.6ns 0.314 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 412ns 0.354ns 1.32ns 0.00815 0 0 584 B
master StartFinishSpan netcoreapp3.1 582ns 0.765ns 2.96ns 0.00573 0 0 584 B
master StartFinishSpan net472 658ns 0.658ns 2.37ns 0.0922 0 0 586 B
master StartFinishScope net6.0 492ns 0.316ns 1.18ns 0.00975 0 0 704 B
master StartFinishScope netcoreapp3.1 752ns 0.69ns 2.58ns 0.00755 0 0 704 B
master StartFinishScope net472 838ns 0.778ns 3.01ns 0.105 0 0 666 B
#6892 StartFinishSpan net6.0 402ns 0.28ns 1.05ns 0.00805 0 0 584 B
#6892 StartFinishSpan netcoreapp3.1 601ns 0.626ns 2.34ns 0.00641 0 0 584 B
#6892 StartFinishSpan net472 688ns 0.476ns 1.84ns 0.0898 0 0 586 B
#6892 StartFinishScope net6.0 496ns 0.479ns 1.85ns 0.00764 0 0 704 B
#6892 StartFinishScope netcoreapp3.1 724ns 0.502ns 1.81ns 0.00726 0 0 704 B
#6892 StartFinishScope net472 901ns 0.514ns 1.92ns 0.105 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6892

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.123 597.07 670.33

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 597ns 0.475ns 1.84ns 0.00901 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1μs 1.28ns 4.78ns 0.00501 0 0 704 B
master RunOnMethodBegin net472 1.14μs 1.45ns 5.62ns 0.102 0 0 666 B
#6892 RunOnMethodBegin net6.0 670ns 0.676ns 2.62ns 0.0101 0 0 704 B
#6892 RunOnMethodBegin netcoreapp3.1 923ns 1.42ns 5.49ns 0.0094 0 0 704 B
#6892 RunOnMethodBegin net472 1.15μs 0.39ns 1.51ns 0.104 0 0 666 B

@andrewlock
Copy link
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6892) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6892) - mean (1,010ms)  : 987, 1034
     .   : milestone, 1010,
    master - mean (1,007ms)  : 987, 1027
     .   : milestone, 1007,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6892) - mean (103ms)  : 100, 105
     .   : milestone, 103,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6892) - mean (697ms)  : 682, 712
     .   : milestone, 697,
    master - mean (694ms)  : 671, 717
     .   : milestone, 694,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6892) - mean (89ms)  : 88, 91
     .   : milestone, 89,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6892) - mean (651ms)  : 631, 672
     .   : milestone, 651,
    master - mean (657ms)  : 632, 683
     .   : milestone, 657,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6892) - mean (191ms)  : 187, 194
     .   : milestone, 191,
    master - mean (189ms)  : 186, 193
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (6892) - mean (1,110ms)  : 1084, 1136
     .   : milestone, 1110,
    master - mean (1,104ms)  : 1075, 1133
     .   : milestone, 1104,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6892) - mean (269ms)  : 265, 273
     .   : milestone, 269,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6892) - mean (879ms)  : 849, 908
     .   : milestone, 879,
    master - mean (882ms)  : 854, 910
     .   : milestone, 882,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6892) - mean (262ms)  : 258, 265
     .   : milestone, 262,
    master - mean (260ms)  : 257, 264
     .   : milestone, 260,

    section CallTarget+Inlining+NGEN
    This PR (6892) - mean (867ms)  : 836, 897
     .   : milestone, 867,
    master - mean (870ms)  : 842, 899
     .   : milestone, 870,
Loading

andrewlock
andrewlock approved these changes Apr 24, 2025
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job 👍

@NachoEchevarria NachoEchevarria marked this pull request as ready for review April 24, 2025 14:44
@NachoEchevarria NachoEchevarria requested review from a team as code owners April 24, 2025 14:44
@NachoEchevarria NachoEchevarria merged commit d758f12 into master Apr 24, 2025
126 of 127 checks passed
@NachoEchevarria NachoEchevarria deleted the nacho/CouchBaseExceptions branch April 24, 2025 14:56
@github-actions github-actions bot added this to the vNext-v3 milestone Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bouwkast bouwkast bouwkast approved these changes

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:tests unit tests, integration tests
Projects
None yet
Milestone
3.16.0
Development

Successfully merging this pull request may close these issues.
3 participants
@NachoEchevarria @andrewlock @bouwkast