Skip to content

Remove throughput tests #6790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 6, 2025
Merged

Remove throughput tests #6790

merged 6 commits into from
May 6, 2025

Conversation

andrewlock
Copy link
Member

@andrewlock andrewlock commented Mar 20, 2025 •
edited
Loading

Summary of changes

Remove the crank throughput tests

Reason for change

The crank throughput tests have been superseded by the macrobenchmarking. We can therefore remove all the crank throughput test infrastructure

Implementation details

  • Delete the pipeline stages
  • Delete the crank configuration files
  • Delete the "throughput comparison" Nuke targets

Test coverage

This is the test

Other details

Warning

We should not merge this until we're sure the macrobenchmarks are fulfilling our requirements. Once we're happy, we can pull the trigger and remove the VMs too.

https://datadoghq.atlassian.net/browse/LANGPLAT-407

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Mar 20, 2025
@andrewlock andrewlock requested a review from a team as a code owner March 20, 2025 15:36
@andrewlock andrewlock added the status:do-not-merge Work is done. Can review, but do not merge yet. label Mar 20, 2025
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Mar 20, 2025 •
edited
Loading

Datadog Report

Branch report: andrew/ci/remove-throughput-tests
Commit report: 1b7b67c
Test service: dd-trace-dotnet

✅ 0 Failed, 250331 Passed, 2263 Skipped, 18h 1m 9.61s Total Time

@andrewlock
Copy link
Member Author

andrewlock commented Mar 20, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6790) - mean (70ms)  : 66, 74
     .   : milestone, 70,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6790) - mean (1,019ms)  : 987, 1050
     .   : milestone, 1019,
    master - mean (1,010ms)  : 983, 1037
     .   : milestone, 1010,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6790) - mean (103ms)  : 101, 105
     .   : milestone, 103,
    master - mean (103ms)  : 101, 106
     .   : milestone, 103,

    section CallTarget+Inlining+NGEN
    This PR (6790) - mean (694ms)  : 672, 716
     .   : milestone, 694,
    master - mean (695ms)  : 679, 711
     .   : milestone, 695,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6790) - mean (90ms)  : 88, 92
     .   : milestone, 90,
    master - mean (89ms)  : 88, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6790) - mean (650ms)  : 633, 668
     .   : milestone, 650,
    master - mean (653ms)  : 634, 672
     .   : milestone, 653,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6790) - mean (190ms)  : 186, 195
     .   : milestone, 190,
    master - mean (191ms)  : 187, 195
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6790) - mean (1,113ms)  : 1084, 1141
     .   : milestone, 1113,
    master - mean (1,113ms)  : 1083, 1144
     .   : milestone, 1113,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6790) - mean (270ms)  : 265, 275
     .   : milestone, 270,
    master - mean (271ms)  : 265, 277
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (6790) - mean (882ms)  : 850, 915
     .   : milestone, 882,
    master - mean (880ms)  : 857, 904
     .   : milestone, 880,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6790) - mean (262ms)  : 258, 266
     .   : milestone, 262,
    master - mean (262ms)  : 258, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6790) - mean (862ms)  : 835, 889
     .   : milestone, 862,
    master - mean (869ms)  : 843, 895
     .   : milestone, 869,
Loading

@andrewlock
Copy link
Member Author

andrewlock commented Mar 20, 2025 •
edited
Loading

Benchmarks Report for tracer 🐌

Benchmarks for #6790 compared to master:

  • 3 benchmarks are slower, with geometric mean 1.175
  • 1 benchmarks have fewer allocations
  • 3 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6790

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.71 KB 5.79 KB 80 B 1.40%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.13 KB 6.17 KB 37 B 0.60%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.81μs 118ns 1.18μs 0 0 0 5.57 KB
master StartStopWithChild netcoreapp3.1 10.4μs 100ns 970ns 0 0 0 5.71 KB
master StartStopWithChild net472 15.1μs 85.3ns 597ns 0.965 0.322 0 6.13 KB
#6790 StartStopWithChild net6.0 8.57μs 149ns 1.49μs 0 0 0 5.57 KB
#6790 StartStopWithChild netcoreapp3.1 10μs 112ns 1.11μs 0 0 0 5.79 KB
#6790 StartStopWithChild net472 15.6μs 88.4ns 612ns 1.02 0.293 0.0732 6.17 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 513μs 502ns 1.81μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 655μs 937ns 3.51μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 849μs 651ns 2.44μs 0 0 0 3.35 KB
#6790 WriteAndFlushEnrichedTraces net6.0 490μs 568ns 2.13μs 0 0 0 2.7 KB
#6790 WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 2.52μs 9.09μs 0 0 0 2.7 KB
#6790 WriteAndFlushEnrichedTraces net472 875μs 587ns 2.27μs 0 0 0 3.35 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 136μs 746ns 4.35μs 0 0 0 14.48 KB
master SendRequest netcoreapp3.1 143μs 783ns 4.64μs 0 0 0 17.28 KB
master SendRequest net472 0.00219ns 0.000947ns 0.00367ns 0 0 0 0 b
#6790 SendRequest net6.0 130μs 744ns 5.42μs 0 0 0 14.48 KB
#6790 SendRequest netcoreapp3.1 152μs 755ns 3.46μs 0 0 0 17.28 KB
#6790 SendRequest net472 0.00309ns 0.00135ns 0.00488ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6790

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.61 KB 42.01 KB 407 B 0.98%

Fewer allocations 🎉 in #6790

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 42.52 KB 41.74 KB -773 B -1.82%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 596μs 5.12μs 49.9μs 0 0 0 42.52 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 689μs 5.32μs 52.1μs 0 0 0 41.61 KB
master WriteAndFlushEnrichedTraces net472 891μs 5.18μs 44μs 8.33 4.17 0 53.42 KB
#6790 WriteAndFlushEnrichedTraces net6.0 566μs 3.3μs 31.4μs 0 0 0 41.74 KB
#6790 WriteAndFlushEnrichedTraces netcoreapp3.1 702μs 5.78μs 56.1μs 0 0 0 42.01 KB
#6790 WriteAndFlushEnrichedTraces net472 879μs 4.93μs 45.5μs 8.33 4.17 0 53.56 KB
Benchmarks.Trace.DbCommandBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6790

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0 1.165 1,232.32 1,435.83

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.24μs 4.21ns 16.3ns 0.0123 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 1.82μs 5.75ns 22.3ns 0.00895 0 0 1.02 KB
master ExecuteNonQuery net472 2.2μs 5.91ns 21.3ns 0.152 0.0109 0 995 B
#6790 ExecuteNonQuery net6.0 1.44μs 2.7ns 10.4ns 0.0142 0 0 1.03 KB
#6790 ExecuteNonQuery netcoreapp3.1 1.84μs 5.47ns 21.2ns 0.00914 0 0 1.02 KB
#6790 ExecuteNonQuery net472 2.16μs 2.35ns 8.14ns 0.152 0.0108 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.23μs 1.04ns 4.04ns 0.0125 0 0 984 B
master CallElasticsearch netcoreapp3.1 1.6μs 3.33ns 12.9ns 0.00797 0 0 984 B
master CallElasticsearch net472 2.6μs 1.13ns 4.39ns 0.156 0 0 1 KB
master CallElasticsearchAsync net6.0 1.21μs 0.868ns 3.25ns 0.0122 0 0 960 B
master CallElasticsearchAsync netcoreapp3.1 1.73μs 3.31ns 12.4ns 0.00861 0 0 1.03 KB
master CallElasticsearchAsync net472 2.69μs 0.862ns 3.34ns 0.162 0 0 1.06 KB
#6790 CallElasticsearch net6.0 1.15μs 0.881ns 3.3ns 0.0115 0 0 984 B
#6790 CallElasticsearch netcoreapp3.1 1.54μs 1.77ns 6.38ns 0.00759 0 0 984 B
#6790 CallElasticsearch net472 2.67μs 1.82ns 7.05ns 0.159 0 0 1 KB
#6790 CallElasticsearchAsync net6.0 1.29μs 0.72ns 2.69ns 0.0128 0 0 960 B
#6790 CallElasticsearchAsync netcoreapp3.1 1.7μs 1.23ns 4.6ns 0.00849 0 0 1.03 KB
#6790 CallElasticsearchAsync net472 2.63μs 1.05ns 3.78ns 0.158 0 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.35μs 0.767ns 2.87ns 0.0136 0 0 960 B
master ExecuteAsync netcoreapp3.1 1.67μs 2.07ns 7.75ns 0.00829 0 0 960 B
master ExecuteAsync net472 1.85μs 1.24ns 4.8ns 0.138 0 0 923 B
#6790 ExecuteAsync net6.0 1.26μs 2.01ns 7.8ns 0.0122 0 0 960 B
#6790 ExecuteAsync netcoreapp3.1 1.7μs 1.77ns 6.84ns 0.0085 0 0 960 B
#6790 ExecuteAsync net472 1.81μs 1.19ns 4.6ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.29μs 3.22ns 12.5ns 0.0215 0 0 2.32 KB
master SendAsync netcoreapp3.1 5.29μs 7.05ns 27.3ns 0.0264 0 0 2.86 KB
master SendAsync net472 7.54μs 6.44ns 25ns 0.486 0 0 3.13 KB
#6790 SendAsync net6.0 4.34μs 2.65ns 10.3ns 0.0218 0 0 2.32 KB
#6790 SendAsync netcoreapp3.1 5.32μs 4.64ns 17.4ns 0.0265 0 0 2.86 KB
#6790 SendAsync net472 7.39μs 4.27ns 16ns 0.482 0 0 3.13 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.7μs 8.99ns 46.7ns 0.0168 0 0 1.71 KB
master EnrichedLog netcoreapp3.1 2.32μs 1.52ns 5.47ns 0.0232 0 0 1.71 KB
master EnrichedLog net472 2.64μs 1.42ns 5.51ns 0.249 0 0 1.64 KB
#6790 EnrichedLog net6.0 1.55μs 1.86ns 6.95ns 0.0231 0 0 1.71 KB
#6790 EnrichedLog netcoreapp3.1 2.36μs 1.11ns 4.02ns 0.0118 0 0 1.71 KB
#6790 EnrichedLog net472 2.9μs 3.03ns 11.7ns 0.253 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 350ns 1.31μs 0 0 0 4.32 KB
master EnrichedLog netcoreapp3.1 115μs 302ns 1.17μs 0 0 0 4.32 KB
master EnrichedLog net472 149μs 160ns 621ns 0 0 0 4.51 KB
#6790 EnrichedLog net6.0 113μs 155ns 601ns 0 0 0 4.32 KB
#6790 EnrichedLog netcoreapp3.1 116μs 116ns 433ns 0 0 0 4.32 KB
#6790 EnrichedLog net472 154μs 476ns 1.72μs 0 0 0 4.51 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.01μs 1.65ns 6.38ns 0.0302 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 4.11μs 1.68ns 6.06ns 0.0204 0 0 2.26 KB
master EnrichedLog net472 4.92μs 2.85ns 11ns 0.319 0 0 2.09 KB
#6790 EnrichedLog net6.0 3.24μs 1.2ns 4.34ns 0.0325 0 0 2.26 KB
#6790 EnrichedLog netcoreapp3.1 4.34μs 2.93ns 11ns 0.0216 0 0 2.26 KB
#6790 EnrichedLog net472 4.86μs 2.8ns 10.8ns 0.317 0 0 2.09 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.39μs 0.669ns 2.59ns 0.0139 0 0 1.15 KB
master SendReceive netcoreapp3.1 1.78μs 2.56ns 9.56ns 0.00889 0 0 1.15 KB
master SendReceive net472 2.1μs 0.675ns 2.61ns 0.179 0 0 1.16 KB
#6790 SendReceive net6.0 1.31μs 1.3ns 5.05ns 0.0131 0 0 1.15 KB
#6790 SendReceive netcoreapp3.1 1.76μs 1.65ns 6.19ns 0.00888 0 0 1.15 KB
#6790 SendReceive net472 2.13μs 9.74ns 37.7ns 0.175 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.68μs 2.13ns 7.67ns 0.0134 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 3.84μs 3.76ns 13.5ns 0.0191 0 0 1.69 KB
master EnrichedLog net472 4.44μs 2.15ns 7.74ns 0.311 0 0 2.08 KB
#6790 EnrichedLog net6.0 2.83μs 1.72ns 6.66ns 0.0141 0 0 1.64 KB
#6790 EnrichedLog netcoreapp3.1 4.03μs 7.26ns 28.1ns 0.0199 0 0 1.69 KB
#6790 EnrichedLog net472 4.5μs 3.47ns 13.4ns 0.314 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6790

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 1.192 776.93 926.06
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.167 407.58 475.80

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 408ns 0.29ns 1.12ns 0.00818 0 0 584 B
master StartFinishSpan netcoreapp3.1 614ns 0.931ns 3.6ns 0.0062 0 0 584 B
master StartFinishSpan net472 629ns 0.277ns 1.04ns 0.0912 0 0 586 B
master StartFinishScope net6.0 528ns 0.477ns 1.79ns 0.00793 0 0 704 B
master StartFinishScope netcoreapp3.1 779ns 0.583ns 2.18ns 0.00778 0 0 704 B
master StartFinishScope net472 777ns 0.343ns 1.28ns 0.105 0 0 666 B
#6790 StartFinishSpan net6.0 476ns 0.314ns 1.22ns 0.00713 0 0 584 B
#6790 StartFinishSpan netcoreapp3.1 570ns 0.461ns 1.6ns 0.00569 0 0 584 B
#6790 StartFinishSpan net472 604ns 0.266ns 0.996ns 0.0912 0 0 586 B
#6790 StartFinishScope net6.0 543ns 0.227ns 0.817ns 0.00814 0 0 704 B
#6790 StartFinishScope netcoreapp3.1 754ns 0.741ns 2.77ns 0.0076 0 0 704 B
#6790 StartFinishScope net472 926ns 0.312ns 1.13ns 0.105 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 639ns 0.54ns 2.02ns 0.00687 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 934ns 0.89ns 3.21ns 0.00929 0 0 704 B
master RunOnMethodBegin net472 1.12μs 0.785ns 2.94ns 0.101 0 0 666 B
#6790 RunOnMethodBegin net6.0 584ns 0.641ns 2.48ns 0.00871 0 0 704 B
#6790 RunOnMethodBegin netcoreapp3.1 898ns 0.99ns 3.83ns 0.00874 0 0 704 B
#6790 RunOnMethodBegin net472 1.11μs 0.624ns 2.33ns 0.1 0 0 666 B

@andrewlock andrewlock mentioned this pull request Apr 23, 2025
Merged
andrewlock added a commit that referenced this pull request Apr 23, 2025
@andrewlock
Don't run throughput tests on PRs (#6884)
48ee053 
## Summary of changes

Don't run the old throughput tests on PRs

## Reason for change

- We haven't had a successful run for a week
- We're migrating to the microbenchmark platform

## Implementation details

Don't run by default on PRs (though allow manual triggering). I haven't
_completely_ deleted them yet (that's done in
#6790) because we're not
completely happy with the results from the microbenchmark platform, and
so we're still using them to gate the release

## Test coverage

This is the test. As long as the throughput tests don't run, I think
we're fine
@andrewlock andrewlock force-pushed the andrew/ci/remove-throughput-tests branch from a2587c3 to 1b7b67c Compare May 6, 2025 12:01
@andrewlock andrewlock removed the status:do-not-merge Work is done. Can review, but do not merge yet. label May 6, 2025
gleocadie
gleocadie approved these changes May 6, 2025
View reviewed changes
Copy link
Collaborator

@gleocadie gleocadie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

NachoEchevarria
NachoEchevarria approved these changes May 6, 2025
View reviewed changes
Copy link
Contributor

@NachoEchevarria NachoEchevarria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@pr-commenter
Copy link

pr-commenter bot commented May 6, 2025

Benchmarks

Benchmarks Report for tracer 🐌

Benchmarks for #6790 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.133
  • 2 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.8μs 55.5ns 266ns 0 0 0 5.54 KB
master StartStopWithChild netcoreapp3.1 13.3μs 65.2ns 292ns 0 0 0 5.76 KB
master StartStopWithChild net472 21.2μs 87.8ns 383ns 0.994 0.331 0.11 6.12 KB
#6790 StartStopWithChild net6.0 10.5μs 57.3ns 319ns 0 0 0 5.56 KB
#6790 StartStopWithChild netcoreapp3.1 13.3μs 67.5ns 324ns 0 0 0 5.75 KB
#6790 StartStopWithChild net472 22.1μs 125ns 868ns 0.983 0.328 0.109 6.12 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 945μs 35.6ns 123ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 163ns 588ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.22ms 41.6ns 144ns 0 0 0 3.31 KB
#6790 WriteAndFlushEnrichedTraces net6.0 929μs 130ns 468ns 0 0 0 2.71 KB
#6790 WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 301ns 1.17μs 0 0 0 2.7 KB
#6790 WriteAndFlushEnrichedTraces net472 1.22ms 97.4ns 377ns 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61.3μs 59.3ns 230ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 71.3μs 89.3ns 322ns 0 0 0 17.42 KB
master SendRequest net472 0.000237ns 0.000237ns 0.00092ns 0 0 0 0 b
#6790 SendRequest net6.0 60.6μs 69.4ns 250ns 0 0 0 14.53 KB
#6790 SendRequest netcoreapp3.1 70.6μs 44.3ns 160ns 0 0 0 17.42 KB
#6790 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 Fewer allocations 🎉

Faster 🎉 in #6790

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 1.133 715,163.32 631,104.30

Fewer allocations 🎉 in #6790

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 42.13 KB 41.82 KB -306 B -0.73%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 56.7 KB 55.82 KB -877 B -1.55%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 705μs 3.72μs 18.2μs 0 0 0 41.86 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 731μs 4.12μs 28.2μs 0 0 0 42.13 KB
master WriteAndFlushEnrichedTraces net472 1.06ms 6.19μs 61.3μs 8.33 0 0 56.7 KB
#6790 WriteAndFlushEnrichedTraces net6.0 630μs 492ns 1.84μs 0 0 0 41.9 KB
#6790 WriteAndFlushEnrichedTraces netcoreapp3.1 679μs 3.78μs 22.7μs 0 0 0 41.82 KB
#6790 WriteAndFlushEnrichedTraces net472 1.04ms 4.98μs 21.7μs 4.81 0 0 55.82 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 2.06μs 10.3ns 41.3ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.53μs 8.46ns 32.8ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.85μs 4.34ns 16.8ns 0.155 0.0141 0 995 B
#6790 ExecuteNonQuery net6.0 1.94μs 2.81ns 10.9ns 0 0 0 1.03 KB
#6790 ExecuteNonQuery netcoreapp3.1 2.52μs 1.06ns 3.65ns 0 0 0 1.02 KB
#6790 ExecuteNonQuery net472 2.78μs 2.18ns 8.44ns 0.153 0.0139 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.82μs 4.7ns 18.2ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.36μs 8.48ns 31.7ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.59μs 3.55ns 13.3ns 0.163 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.84μs 6.69ns 25.9ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.42μs 9.5ns 36.8ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.68μs 4.18ns 16.2ns 0.165 0 0 1.11 KB
#6790 CallElasticsearch net6.0 1.85μs 10.1ns 56ns 0 0 0 1.04 KB
#6790 CallElasticsearch netcoreapp3.1 2.33μs 4.97ns 18.6ns 0 0 0 1.04 KB
#6790 CallElasticsearch net472 3.56μs 3.62ns 13.6ns 0.159 0 0 1.05 KB
#6790 CallElasticsearchAsync net6.0 1.83μs 9.07ns 37.4ns 0 0 0 1.02 KB
#6790 CallElasticsearchAsync netcoreapp3.1 2.43μs 6.25ns 24.2ns 0 0 0 1.09 KB
#6790 CallElasticsearchAsync net472 3.72μs 1.93ns 7.22ns 0.168 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.88μs 0.634ns 2.37ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.29μs 9.93ns 38.5ns 0 0 0 960 B
master ExecuteAsync net472 2.6μs 1.85ns 7.17ns 0.143 0 0 923 B
#6790 ExecuteAsync net6.0 1.93μs 2.79ns 10.8ns 0 0 0 960 B
#6790 ExecuteAsync netcoreapp3.1 2.23μs 10.8ns 43.4ns 0 0 0 960 B
#6790 ExecuteAsync net472 2.62μs 1.52ns 5.88ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.83μs 5.16ns 19.3ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.32μs 14.5ns 56.3ns 0 0 0 2.9 KB
master SendAsync net472 12.3μs 6.94ns 26ns 0.488 0 0 3.19 KB
#6790 SendAsync net6.0 7.03μs 3.22ns 12.1ns 0 0 0 2.37 KB
#6790 SendAsync netcoreapp3.1 8.76μs 20ns 77.5ns 0 0 0 2.9 KB
#6790 SendAsync net472 12.3μs 9.88ns 37ns 0.494 0 0 3.19 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.58μs 1.52ns 5.9ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.51μs 2.14ns 7.73ns 0 0 0 1.76 KB
master EnrichedLog net472 3.92μs 4.04ns 15.6ns 0.255 0 0 1.69 KB
#6790 EnrichedLog net6.0 2.55μs 12.2ns 51.7ns 0 0 0 1.76 KB
#6790 EnrichedLog netcoreapp3.1 3.55μs 2.91ns 11.3ns 0 0 0 1.76 KB
#6790 EnrichedLog net472 3.88μs 4.02ns 15.6ns 0.252 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 122μs 64.1ns 222ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 128μs 68ns 255ns 0 0 0 4.37 KB
master EnrichedLog net472 166μs 202ns 781ns 0 0 0 4.57 KB
#6790 EnrichedLog net6.0 128μs 552ns 2.21μs 0 0 0 4.37 KB
#6790 EnrichedLog netcoreapp3.1 131μs 457ns 1.65μs 0 0 0 4.37 KB
#6790 EnrichedLog net472 172μs 402ns 1.5μs 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.13μs 20.3ns 78.8ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.56μs 19.2ns 71.7ns 0 0 0 2.32 KB
master EnrichedLog net472 7.34μs 4.84ns 18.7ns 0.331 0 0 2.14 KB
#6790 EnrichedLog net6.0 4.96μs 6.24ns 24.2ns 0 0 0 2.32 KB
#6790 EnrichedLog netcoreapp3.1 6.76μs 22.1ns 85.7ns 0 0 0 2.32 KB
#6790 EnrichedLog net472 7.39μs 5.54ns 21.4ns 0.334 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.05μs 10.8ns 54ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.62μs 6.95ns 26.9ns 0 0 0 1.21 KB
master SendReceive net472 3.11μs 2.64ns 9.9ns 0.184 0 0 1.21 KB
#6790 SendReceive net6.0 2.01μs 2.46ns 9.52ns 0 0 0 1.21 KB
#6790 SendReceive netcoreapp3.1 2.61μs 7.47ns 26.9ns 0 0 0 1.21 KB
#6790 SendReceive net472 2.98μs 1.41ns 5.48ns 0.179 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.2μs 4.87ns 18.2ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.81μs 25.7ns 99.4ns 0 0 0 1.69 KB
master EnrichedLog net472 6.96μs 7.13ns 27.6ns 0.31 0 0 2.08 KB
#6790 EnrichedLog net6.0 4.32μs 16.6ns 64.5ns 0 0 0 1.64 KB
#6790 EnrichedLog netcoreapp3.1 5.53μs 13.2ns 51.2ns 0 0 0 1.69 KB
#6790 EnrichedLog net472 6.56μs 6.35ns 24.6ns 0.328 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 759ns 3.81ns 17.5ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 945ns 4.59ns 18.3ns 0 0 0 584 B
master StartFinishSpan net472 904ns 1.72ns 6.67ns 0.0923 0 0 586 B
master StartFinishScope net6.0 913ns 4.55ns 18.8ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.18μs 1.54ns 5.95ns 0 0 0 704 B
master StartFinishScope net472 1.13μs 0.412ns 1.54ns 0.102 0 0 666 B
#6790 StartFinishSpan net6.0 764ns 0.593ns 2.22ns 0 0 0 584 B
#6790 StartFinishSpan netcoreapp3.1 942ns 4.52ns 18.1ns 0 0 0 584 B
#6790 StartFinishSpan net472 892ns 0.6ns 2.32ns 0.0895 0 0 586 B
#6790 StartFinishScope net6.0 892ns 4.87ns 26.7ns 0 0 0 704 B
#6790 StartFinishScope netcoreapp3.1 1.15μs 5.9ns 30.7ns 0 0 0 704 B
#6790 StartFinishScope net472 1.08μs 0.738ns 2.86ns 0.102 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.05μs 1.2ns 4.65ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.32μs 6.76ns 30.2ns 0 0 0 704 B
master RunOnMethodBegin net472 1.36μs 1.16ns 4.48ns 0.102 0 0 666 B
#6790 RunOnMethodBegin net6.0 1.05μs 5.17ns 21.3ns 0 0 0 704 B
#6790 RunOnMethodBegin netcoreapp3.1 1.4μs 7.33ns 38.1ns 0 0 0 704 B
#6790 RunOnMethodBegin net472 1.36μs 3.49ns 13.5ns 0.101 0 0 666 B

@andrewlock andrewlock merged commit 871b078 into master May 6, 2025
128 checks passed
@andrewlock andrewlock deleted the andrew/ci/remove-throughput-tests branch May 6, 2025 14:51
@github-actions github-actions bot added this to the vNext-v3 milestone May 6, 2025
@andrewlock
Copy link
Member Author

We're not actually entirely happy with the tests we have, but Crank has extensive issues and has essentially stopped running, so just removed it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bouwkast bouwkast bouwkast approved these changes

@kevingosse kevingosse kevingosse approved these changes

@gleocadie gleocadie gleocadie approved these changes

@NachoEchevarria NachoEchevarria NachoEchevarria approved these changes

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.17.0
Development

Successfully merging this pull request may close these issues.
5 participants
@andrewlock @bouwkast @kevingosse @gleocadie @NachoEchevarria