Skip to content

[Benchmarks] Update microbenchmarks #6868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Apr 17, 2025
Merged

[Benchmarks] Update microbenchmarks #6868

merged 5 commits into from
Apr 17, 2025

Conversation

faydef
Copy link
Contributor

@faydef faydef commented Apr 17, 2025 •
edited
Loading

Summary of changes

This PR updates the microbenchmarks:

  • The microbenchmarks now run a new fixed version of the tooling
  • Benchmarks run on every commit and publish results only on PRs
  • When a job is cancelled, the spawned infrastructure is deleted (otherwise it hangs and creates extra cost before scheduled cleanup), example of cancelled job with destruction of spawned infrastructure
  • Uses a more conventional naming for branch

Reason for change

  • Noticed some extra unplanned costs.
  • Need to update tooling version

Implementation details

Test coverage

Other details

@faydef faydef changed the title Fayssal/update microbenchmarks [Benchmarks] Update microbenchmarks Apr 17, 2025
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Apr 17, 2025 •
edited
Loading

Datadog Report

Branch report: fayssal/update-microbenchmarks
Commit report: bd5ab05
Test service: dd-trace-dotnet

✅ 0 Failed, 258415 Passed, 2430 Skipped, 20h 22m 5.89s Total Time

@pr-commenter
Copy link

pr-commenter bot commented Apr 17, 2025

Benchmarks

Benchmarks Report for tracer 🐌

Benchmarks for #99 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.125
  • 2 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.11 KB 6.07 KB -38 B -0.62%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.7μs 57.8ns 322ns 0 0 0 5.54 KB
master StartStopWithChild netcoreapp3.1 13.5μs 65.7ns 263ns 0 0 0 5.74 KB
master StartStopWithChild net472 21.9μs 116ns 581ns 1 0.334 0.111 6.11 KB
#99 StartStopWithChild net6.0 10.3μs 56.5ns 362ns 0 0 0 5.56 KB
#99 StartStopWithChild netcoreapp3.1 13.3μs 66.9ns 307ns 0 0 0 5.73 KB
#99 StartStopWithChild net472 22μs 94.4ns 340ns 0.978 0.435 0.109 6.07 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 3.33 KB 3.31 KB -23 B -0.69%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 951μs 175ns 630ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 529ns 2.05μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.25ms 1.44μs 5.57μs 0 0 0 3.33 KB
#99 WriteAndFlushEnrichedTraces net6.0 929μs 82.6ns 309ns 0 0 0 2.71 KB
#99 WriteAndFlushEnrichedTraces netcoreapp3.1 1.01ms 76.6ns 287ns 0 0 0 2.7 KB
#99 WriteAndFlushEnrichedTraces net472 1.22ms 630ns 2.44μs 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61.6μs 244ns 943ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 70.7μs 65ns 243ns 0 0 0 17.42 KB
master SendRequest net472 0.00936ns 0.00289ns 0.0112ns 0 0 0 0 b
#99 SendRequest net6.0 61.4μs 50.3ns 195ns 0 0 0 14.53 KB
#99 SendRequest netcoreapp3.1 71.2μs 63.8ns 221ns 0 0 0 17.42 KB
#99 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #99

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 1.125 749,961.04 666,915.21

More allocations ⚠️ in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 55.94 KB 56.25 KB 316 B 0.56%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 750μs 516ns 1.93μs 0 0 0 41.86 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 688μs 3.7μs 19.9μs 0 0 0 41.9 KB
master WriteAndFlushEnrichedTraces net472 890μs 3.25μs 12.6μs 4.46 0 0 55.94 KB
#99 WriteAndFlushEnrichedTraces net6.0 667μs 561ns 2.1μs 0 0 0 41.74 KB
#99 WriteAndFlushEnrichedTraces netcoreapp3.1 694μs 3.94μs 35.3μs 0 0 0 41.85 KB
#99 WriteAndFlushEnrichedTraces net472 877μs 4.24μs 18μs 8.33 0 0 56.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 2.03μs 1.54ns 5.57ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.5μs 1.26ns 4.89ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.78μs 2.45ns 9.5ns 0.154 0.014 0 995 B
#99 ExecuteNonQuery net6.0 1.99μs 8.73ns 33.8ns 0 0 0 1.03 KB
#99 ExecuteNonQuery netcoreapp3.1 2.55μs 10.3ns 39.9ns 0 0 0 1.02 KB
#99 ExecuteNonQuery net472 2.84μs 2.03ns 7.86ns 0.156 0.0142 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.84μs 7.46ns 28.9ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.37μs 1.31ns 5.09ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.61μs 6.1ns 23.6ns 0.161 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.83μs 9.27ns 34.7ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.34μs 11.5ns 45.8ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.78μs 7.35ns 28.5ns 0.168 0 0 1.11 KB
#99 CallElasticsearch net6.0 1.82μs 3.66ns 14.2ns 0 0 0 1.04 KB
#99 CallElasticsearch netcoreapp3.1 2.22μs 10.6ns 43.8ns 0 0 0 1.04 KB
#99 CallElasticsearch net472 3.59μs 4.37ns 16.9ns 0.16 0 0 1.05 KB
#99 CallElasticsearchAsync net6.0 1.84μs 8.9ns 37.8ns 0 0 0 1.02 KB
#99 CallElasticsearchAsync netcoreapp3.1 2.36μs 10.9ns 43.7ns 0 0 0 1.09 KB
#99 CallElasticsearchAsync net472 3.68μs 2.45ns 9.47ns 0.167 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.82μs 8.13ns 31.5ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.33μs 9.1ns 32.8ns 0 0 0 960 B
master ExecuteAsync net472 2.61μs 3ns 11.6ns 0.143 0 0 923 B
#99 ExecuteAsync net6.0 1.8μs 6.16ns 23.9ns 0 0 0 960 B
#99 ExecuteAsync netcoreapp3.1 2.29μs 5.69ns 21.3ns 0 0 0 960 B
#99 ExecuteAsync net472 2.57μs 1.27ns 4.74ns 0.14 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.8μs 2.45ns 8.83ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.36μs 16.4ns 61.5ns 0 0 0 2.9 KB
master SendAsync net472 12.3μs 10.7ns 41.4ns 0.489 0 0 3.19 KB
#99 SendAsync net6.0 7.14μs 18.9ns 70.7ns 0 0 0 2.37 KB
#99 SendAsync netcoreapp3.1 8.52μs 5.91ns 22.9ns 0 0 0 2.9 KB
#99 SendAsync net472 12.4μs 10.8ns 42ns 0.499 0 0 3.19 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.64μs 13.2ns 60.7ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.45μs 16.4ns 63.7ns 0 0 0 1.76 KB
master EnrichedLog net472 3.91μs 4.41ns 17.1ns 0.254 0 0 1.69 KB
#99 EnrichedLog net6.0 2.61μs 13.4ns 67ns 0 0 0 1.76 KB
#99 EnrichedLog netcoreapp3.1 3.37μs 17.5ns 80.1ns 0 0 0 1.76 KB
#99 EnrichedLog net472 3.9μs 2.14ns 8.3ns 0.253 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 122μs 50.6ns 189ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 127μs 201ns 777ns 0 0 0 4.37 KB
master EnrichedLog net472 166μs 222ns 862ns 0 0 0 4.57 KB
#99 EnrichedLog net6.0 124μs 117ns 421ns 0 0 0 4.37 KB
#99 EnrichedLog netcoreapp3.1 129μs 160ns 618ns 0 0 0 4.37 KB
#99 EnrichedLog net472 170μs 163ns 632ns 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.05μs 22.1ns 85.7ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.66μs 23.3ns 90.3ns 0 0 0 2.32 KB
master EnrichedLog net472 7.33μs 7.92ns 30.7ns 0.328 0 0 2.14 KB
#99 EnrichedLog net6.0 4.91μs 12.1ns 46.7ns 0 0 0 2.32 KB
#99 EnrichedLog netcoreapp3.1 6.69μs 6.63ns 25.7ns 0 0 0 2.32 KB
#99 EnrichedLog net472 7.33μs 7.26ns 28.1ns 0.329 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.03μs 10.6ns 49.9ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.85μs 13.4ns 57ns 0 0 0 1.21 KB
master SendReceive net472 3.2μs 3.07ns 11.9ns 0.192 0 0 1.21 KB
#99 SendReceive net6.0 1.99μs 9.11ns 35.3ns 0 0 0 1.21 KB
#99 SendReceive netcoreapp3.1 2.69μs 7.15ns 27.7ns 0 0 0 1.21 KB
#99 SendReceive net472 3.12μs 5.82ns 22.5ns 0.186 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.28μs 3.92ns 13.6ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.69μs 21.9ns 82.1ns 0 0 0 1.69 KB
master EnrichedLog net472 6.67μs 9.04ns 35ns 0.3 0 0 2.08 KB
#99 EnrichedLog net6.0 4.26μs 9.94ns 37.2ns 0 0 0 1.64 KB
#99 EnrichedLog netcoreapp3.1 5.66μs 16.9ns 65.5ns 0 0 0 1.69 KB
#99 EnrichedLog net472 6.71μs 5.66ns 21.9ns 0.3 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 756ns 3.87ns 18.6ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 954ns 5.04ns 25.2ns 0 0 0 584 B
master StartFinishSpan net472 914ns 0.566ns 2.19ns 0.0917 0 0 586 B
master StartFinishScope net6.0 887ns 5.02ns 33.3ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.16μs 5.54ns 23.5ns 0 0 0 704 B
master StartFinishScope net472 1.11μs 0.388ns 1.5ns 0.105 0 0 666 B
#99 StartFinishSpan net6.0 742ns 4.13ns 27.4ns 0 0 0 584 B
#99 StartFinishSpan netcoreapp3.1 946ns 4.99ns 24.4ns 0 0 0 584 B
#99 StartFinishSpan net472 923ns 0.268ns 1.04ns 0.0927 0 0 586 B
#99 StartFinishScope net6.0 917ns 4.57ns 19.9ns 0 0 0 704 B
#99 StartFinishScope netcoreapp3.1 1.17μs 6.23ns 33.5ns 0 0 0 704 B
#99 StartFinishScope net472 1.11μs 1.55ns 6.02ns 0.105 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.04μs 5.46ns 28.4ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.33μs 4.94ns 19.1ns 0 0 0 704 B
master RunOnMethodBegin net472 1.48μs 0.58ns 2.25ns 0.104 0 0 666 B
#99 RunOnMethodBegin net6.0 1.05μs 5.7ns 32.3ns 0 0 0 704 B
#99 RunOnMethodBegin netcoreapp3.1 1.33μs 6.96ns 32.7ns 0 0 0 704 B
#99 RunOnMethodBegin net472 1.36μs 0.56ns 2.17ns 0.102 0 0 666 B

@andrewlock
Copy link
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6868 compared to master:

  • 3 benchmarks are faster, with geometric mean 1.129
  • 1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.11μs 160ns 1.6μs 0 0 0 5.57 KB
master StartStopWithChild netcoreapp3.1 10.2μs 102ns 986ns 0 0 0 5.75 KB
master StartStopWithChild net472 15.4μs 84.3ns 520ns 1 0.286 0.0716 6.15 KB
#6868 StartStopWithChild net6.0 7.34μs 66.4ns 630ns 0 0 0 5.55 KB
#6868 StartStopWithChild netcoreapp3.1 10.9μs 158ns 1.58μs 0 0 0 5.78 KB
#6868 StartStopWithChild net472 14.8μs 81.9ns 607ns 0.93 0.215 0 6.13 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6868

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 3.35 KB 3.31 KB -34 B -1.02%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 496μs 525ns 1.89μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 662μs 898ns 3.24μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 838μs 555ns 2μs 0 0 0 3.35 KB
#6868 WriteAndFlushEnrichedTraces net6.0 508μs 1.3μs 4.86μs 0 0 0 2.7 KB
#6868 WriteAndFlushEnrichedTraces netcoreapp3.1 704μs 1.31μs 4.73μs 0 0 0 2.7 KB
#6868 WriteAndFlushEnrichedTraces net472 854μs 360ns 1.35μs 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 130μs 651ns 2.98μs 0 0 0 14.48 KB
master SendRequest netcoreapp3.1 145μs 584ns 2.26μs 0 0 0 17.28 KB
master SendRequest net472 0.00213ns 0.000956ns 0.0037ns 0 0 0 0 b
#6868 SendRequest net6.0 129μs 648ns 3.11μs 0 0 0 14.48 KB
#6868 SendRequest netcoreapp3.1 143μs 781ns 4.94μs 0 0 0 17.28 KB
#6868 SendRequest net472 0.00162ns 0.000602ns 0.00233ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 572μs 3.17μs 23.1μs 0 0 0 41.62 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 712μs 8.75μs 84.4μs 0 0 0 41.71 KB
master WriteAndFlushEnrichedTraces net472 903μs 5.9μs 59μs 8.33 4.17 0 53.45 KB
#6868 WriteAndFlushEnrichedTraces net6.0 561μs 3.11μs 18.4μs 0 0 0 41.58 KB
#6868 WriteAndFlushEnrichedTraces netcoreapp3.1 659μs 3.59μs 21.9μs 0 0 0 41.7 KB
#6868 WriteAndFlushEnrichedTraces net472 839μs 4.76μs 33.7μs 8.33 4.17 0 53.45 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.35μs 5.65ns 21.1ns 0.0133 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 1.83μs 4.09ns 15.3ns 0.00917 0 0 1.02 KB
master ExecuteNonQuery net472 2.12μs 5.65ns 21.9ns 0.149 0.0107 0 995 B
#6868 ExecuteNonQuery net6.0 1.3μs 0.889ns 3.33ns 0.0129 0 0 1.03 KB
#6868 ExecuteNonQuery netcoreapp3.1 1.73μs 3.1ns 12ns 0.00864 0 0 1.02 KB
#6868 ExecuteNonQuery net472 2.13μs 2.86ns 10.7ns 0.158 0.0105 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.21μs 1.07ns 4.13ns 0.0121 0 0 984 B
master CallElasticsearch netcoreapp3.1 1.68μs 1.55ns 5.58ns 0.00843 0 0 984 B
master CallElasticsearch net472 2.53μs 1.17ns 4.37ns 0.152 0 0 1 KB
master CallElasticsearchAsync net6.0 1.31μs 0.566ns 2.12ns 0.0131 0 0 960 B
master CallElasticsearchAsync netcoreapp3.1 1.66μs 3.66ns 13.7ns 0.00827 0 0 1.03 KB
master CallElasticsearchAsync net472 2.67μs 1.9ns 7.1ns 0.16 0 0 1.06 KB
#6868 CallElasticsearch net6.0 1.24μs 0.896ns 3.47ns 0.0124 0 0 984 B
#6868 CallElasticsearch netcoreapp3.1 1.58μs 2.77ns 9.99ns 0.00799 0 0 984 B
#6868 CallElasticsearch net472 2.61μs 1.56ns 6.04ns 0.156 0 0 1 KB
#6868 CallElasticsearchAsync net6.0 1.31μs 1.51ns 5.85ns 0.0131 0 0 960 B
#6868 CallElasticsearchAsync netcoreapp3.1 1.74μs 2.01ns 7.51ns 0.00866 0 0 1.03 KB
#6868 CallElasticsearchAsync net472 2.62μs 2.38ns 9.2ns 0.156 0 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.26μs 0.511ns 1.91ns 0.0128 0 0 960 B
master ExecuteAsync netcoreapp3.1 1.76μs 1.26ns 4.73ns 0.00874 0 0 960 B
master ExecuteAsync net472 1.79μs 1.31ns 4.91ns 0.144 0 0 923 B
#6868 ExecuteAsync net6.0 1.29μs 1.59ns 6.15ns 0.0129 0 0 960 B
#6868 ExecuteAsync netcoreapp3.1 1.64μs 1.87ns 7.01ns 0.00805 0 0 960 B
#6868 ExecuteAsync net472 1.8μs 0.892ns 3.34ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.38μs 3.11ns 11.6ns 0.0219 0 0 2.32 KB
master SendAsync netcoreapp3.1 5.35μs 5.07ns 19ns 0.0266 0 0 2.86 KB
master SendAsync net472 7.43μs 6.19ns 24ns 0.481 0 0 3.13 KB
#6868 SendAsync net6.0 4.34μs 16.5ns 70.2ns 0.0216 0 0 2.32 KB
#6868 SendAsync netcoreapp3.1 5.29μs 21.4ns 88.3ns 0.0263 0 0 2.86 KB
#6868 SendAsync net472 7.71μs 36.7ns 147ns 0.493 0 0 3.13 KB
Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6868

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1 1.117 2,400.48 2,149.95

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.56μs 1.67ns 6.47ns 0.0236 0 0 1.71 KB
master EnrichedLog netcoreapp3.1 2.4μs 2.06ns 7.44ns 0.0117 0 0 1.71 KB
master EnrichedLog net472 2.62μs 2.98ns 11.5ns 0.248 0 0 1.64 KB
#6868 EnrichedLog net6.0 1.52μs 1.16ns 4.5ns 0.0225 0 0 1.71 KB
#6868 EnrichedLog netcoreapp3.1 2.15μs 2.01ns 7.52ns 0.0214 0 0 1.71 KB
#6868 EnrichedLog net472 2.64μs 3.07ns 11.9ns 0.26 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 181ns 701ns 0 0 0 4.32 KB
master EnrichedLog netcoreapp3.1 117μs 304ns 1.14μs 0 0 0 4.32 KB
master EnrichedLog net472 152μs 265ns 1.03μs 0 0 0 4.51 KB
#6868 EnrichedLog net6.0 115μs 300ns 1.16μs 0 0 0 4.32 KB
#6868 EnrichedLog netcoreapp3.1 116μs 256ns 990ns 0 0 0 4.32 KB
#6868 EnrichedLog net472 152μs 83.2ns 311ns 0 0 0 4.51 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.09μs 2.14ns 8.3ns 0.0309 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 4.15μs 4.36ns 16.3ns 0.0207 0 0 2.26 KB
master EnrichedLog net472 4.52μs 2.75ns 10.6ns 0.318 0 0 2.09 KB
#6868 EnrichedLog net6.0 3.12μs 1.87ns 7.25ns 0.0312 0 0 2.26 KB
#6868 EnrichedLog netcoreapp3.1 4.14μs 3.17ns 11.9ns 0 0 0 2.26 KB
#6868 EnrichedLog net472 4.86μs 6.88ns 25.8ns 0.317 0 0 2.09 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.43μs 1.05ns 3.8ns 0.0142 0 0 1.15 KB
master SendReceive netcoreapp3.1 1.79μs 3.69ns 14.3ns 0.00878 0 0 1.15 KB
master SendReceive net472 2.09μs 1.63ns 6.09ns 0.177 0 0 1.16 KB
#6868 SendReceive net6.0 1.38μs 0.506ns 1.89ns 0.0138 0 0 1.15 KB
#6868 SendReceive netcoreapp3.1 1.85μs 1.35ns 4.68ns 0.00927 0 0 1.15 KB
#6868 SendReceive net472 2.16μs 1.1ns 4.27ns 0.184 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.79μs 1.47ns 5.29ns 0.014 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 3.96μs 3.53ns 12.7ns 0.0197 0 0 1.69 KB
master EnrichedLog net472 4.4μs 4.71ns 18.2ns 0.329 0 0 2.08 KB
#6868 EnrichedLog net6.0 2.85μs 1.6ns 6ns 0.0143 0 0 1.64 KB
#6868 EnrichedLog netcoreapp3.1 4.05μs 4.66ns 17.5ns 0.0202 0 0 1.69 KB
#6868 EnrichedLog net472 4.35μs 2.58ns 9.3ns 0.326 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6868

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.155 467.84 405.10
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.116 628.67 563.46

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 468ns 0.89ns 3.45ns 0.00702 0 0 584 B
master StartFinishSpan netcoreapp3.1 629ns 0.622ns 2.41ns 0.00633 0 0 584 B
master StartFinishSpan net472 580ns 0.398ns 1.43ns 0.0902 0 0 586 B
master StartFinishScope net6.0 483ns 0.381ns 1.48ns 0.0096 0 0 704 B
master StartFinishScope netcoreapp3.1 780ns 1.25ns 4.85ns 0.00779 0 0 704 B
master StartFinishScope net472 824ns 0.682ns 2.64ns 0.103 0 0 666 B
#6868 StartFinishSpan net6.0 405ns 0.245ns 0.885ns 0.00805 0 0 584 B
#6868 StartFinishSpan netcoreapp3.1 564ns 0.48ns 1.79ns 0.00559 0 0 584 B
#6868 StartFinishSpan net472 637ns 0.281ns 1.05ns 0.0905 0 0 586 B
#6868 StartFinishScope net6.0 475ns 0.329ns 1.23ns 0.00962 0 0 704 B
#6868 StartFinishScope netcoreapp3.1 765ns 1.22ns 4.74ns 0.00764 0 0 704 B
#6868 StartFinishScope net472 818ns 0.297ns 1.15ns 0.102 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 677ns 0.435ns 1.69ns 0.00686 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 980ns 1.61ns 6.25ns 0.00489 0 0 704 B
master RunOnMethodBegin net472 1.08μs 0.366ns 1.37ns 0.104 0 0 666 B
#6868 RunOnMethodBegin net6.0 627ns 0.787ns 2.94ns 0.00925 0 0 704 B
#6868 RunOnMethodBegin netcoreapp3.1 973ns 0.904ns 3.26ns 0.00963 0 0 704 B
#6868 RunOnMethodBegin net472 1.07μs 0.565ns 2.11ns 0.102 0 0 666 B

@andrewlock
Copy link
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6868) - mean (69ms)  : 67, 71
     .   : milestone, 69,
    master - mean (70ms)  : 67, 72
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6868) - mean (1,011ms)  : 987, 1034
     .   : milestone, 1011,
    master - mean (1,014ms)  : 986, 1041
     .   : milestone, 1014,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6868) - mean (103ms)  : 101, 106
     .   : milestone, 103,
    master - mean (104ms)  : 101, 106
     .   : milestone, 104,

    section CallTarget+Inlining+NGEN
    This PR (6868) - mean (697ms)  : 678, 717
     .   : milestone, 697,
    master - mean (701ms)  : 682, 720
     .   : milestone, 701,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6868) - mean (90ms)  : 88, 92
     .   : milestone, 90,
    master - mean (91ms)  : 88, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6868) - mean (660ms)  : 639, 681
     .   : milestone, 660,
    master - mean (660ms)  : 638, 681
     .   : milestone, 660,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6868) - mean (189ms)  : 186, 192
     .   : milestone, 189,
    master - mean (189ms)  : 185, 193
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (6868) - mean (1,105ms)  : 1078, 1133
     .   : milestone, 1105,
    master - mean (1,104ms)  : 1079, 1128
     .   : milestone, 1104,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6868) - mean (268ms)  : 264, 273
     .   : milestone, 268,
    master - mean (269ms)  : 265, 273
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (6868) - mean (879ms)  : 841, 917
     .   : milestone, 879,
    master - mean (881ms)  : 852, 909
     .   : milestone, 881,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6868) - mean (260ms)  : 257, 264
     .   : milestone, 260,
    master - mean (261ms)  : 255, 266
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (6868) - mean (865ms)  : 839, 892
     .   : milestone, 865,
    master - mean (869ms)  : 833, 904
     .   : milestone, 869,
Loading

@datadog-ddstaging
Copy link

Datadog Report

Branch report: fayssal/update-microbenchmarks
Commit report: bd5ab05
Test service: dd-trace-dotnet

✅ 0 Failed, 2 Passed, 0 Skipped, 0s Total Time

@faydef faydef marked this pull request as ready for review April 17, 2025 13:33
@faydef faydef requested a review from a team as a code owner April 17, 2025 13:33
andrewlock
andrewlock approved these changes Apr 17, 2025
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@andrewlock
Copy link
Member

Shouldn't there be a PR comment on here from the benchmarking platform though? 🤔

@faydef faydef merged commit aad3d6f into master Apr 17, 2025
123 of 127 checks passed
@faydef faydef deleted the fayssal/update-microbenchmarks branch April 17, 2025 16:33
@github-actions github-actions bot added this to the vNext-v3 milestone Apr 17, 2025
@bouwkast bouwkast added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Apr 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.15.0
Development

Successfully merging this pull request may close these issues.
3 participants
@faydef @andrewlock @bouwkast