Skip to content

Update Windows to hosted managed DevOps pool #6960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 14, 2025
Merged

Update Windows to hosted managed DevOps pool #6960

merged 7 commits into from
May 14, 2025

Conversation

andrewlock
Copy link
Member

Summary of changes

Move the Windows VMs from a VMSS pool to use a Hosted Managed DevOps Pool

Reason for change

We want to unify the infrastructure hence the move from VMSS to hosted pools

Implementation details

  • Set up a new hosted devops pool (I will document this process in confluence once I'm/we're happy)
  • Reused the same image, so should have identical behaviour otherwise
  • Update ultimate-pipeline to use the new pool

Test coverage

This is the test

Other details

Supersedes

As it's a new pool, we have to manually specify pre-provisioning. I've currently set this to 9-5 UTC with 10VMs but we can tweak that as we see fit. It's currently just to ensure we're getting some pre-provisioning benefit. We can tweak that as we see fit (and can use predictive mode in ~3 weeks time, once Azure has time to analyze our usage)

@andrewlock andrewlock requested a review from a team as a code owner May 12, 2025 13:38
@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label May 12, 2025
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented May 12, 2025 •
edited
Loading

Datadog Summary

✅ Code Quality    ✅ Code Security    ✅ Dependencies

Test Optimization Report

Branch report: andrew/ci/switch-to-managed-devops-pools
Commit report: 674406a
Test service: dd-trace-dotnet

✅ 0 Failed, 231181 Passed, 952 Skipped, 11h 37m 49.97s Total Time

Was this helpful? Give us feedback!

bouwkast
bouwkast approved these changes May 12, 2025
View reviewed changes
Copy link
Collaborator

@bouwkast bouwkast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@pr-commenter
Copy link

pr-commenter bot commented May 12, 2025 •
edited
Loading

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #6960 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.791
  • 2 benchmarks have fewer allocations
  • 5 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6960

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.16 KB 6.21 KB 51 B 0.83%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.74 KB 5.77 KB 31 B 0.54%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.9μs 59ns 339ns 0 0 0 5.54 KB
master StartStopWithChild netcoreapp3.1 13.3μs 70.8ns 354ns 0 0 0 5.74 KB
master StartStopWithChild net472 21.4μs 113ns 679ns 0.924 0.231 0 6.16 KB
#6960 StartStopWithChild net6.0 10.6μs 57.9ns 317ns 0 0 0 5.55 KB
#6960 StartStopWithChild netcoreapp3.1 13.8μs 67.7ns 271ns 0 0 0 5.77 KB
#6960 StartStopWithChild net472 22.2μs 120ns 732ns 0.903 0.338 0 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 955μs 80.3ns 300ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 56.1ns 210ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.24ms 4.73μs 18.3μs 0 0 0 3.31 KB
#6960 WriteAndFlushEnrichedTraces net6.0 924μs 57.7ns 208ns 0 0 0 2.71 KB
#6960 WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 73.4ns 265ns 0 0 0 2.7 KB
#6960 WriteAndFlushEnrichedTraces net472 1.24ms 100ns 388ns 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 328μs 1.13μs 4.36μs 0 0 0 197.06 KB
master AllCycleSimpleBody netcoreapp3.1 478μs 1.56μs 6.05μs 0 0 0 204.77 KB
master AllCycleSimpleBody net472 435μs 129ns 499ns 36.6 2.16 0 236.35 KB
master AllCycleMoreComplexBody net6.0 336μs 1.48μs 5.53μs 0 0 0 200.56 KB
master AllCycleMoreComplexBody netcoreapp3.1 487μs 458ns 1.72μs 0 0 0 208.18 KB
master AllCycleMoreComplexBody net472 443μs 128ns 497ns 36.6 2.16 0 239.87 KB
master ObjectExtractorSimpleBody net6.0 319ns 1.71ns 7.65ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 399ns 2.1ns 10.9ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 302ns 0.029ns 0.108ns 0.0441 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.25μs 4.83ns 17.4ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.97μs 11ns 42.8ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.68μs 6.59ns 25.5ns 0.602 0 0 3.8 KB
#6960 AllCycleSimpleBody net6.0 328μs 176ns 611ns 0 0 0 197.06 KB
#6960 AllCycleSimpleBody netcoreapp3.1 515μs 2.12μs 8.21μs 0 0 0 204.77 KB
#6960 AllCycleSimpleBody net472 439μs 105ns 391ns 36.6 2.16 0 236.35 KB
#6960 AllCycleMoreComplexBody net6.0 344μs 1.79μs 8.79μs 0 0 0 200.56 KB
#6960 AllCycleMoreComplexBody netcoreapp3.1 493μs 1.69μs 6.33μs 0 0 0 208.18 KB
#6960 AllCycleMoreComplexBody net472 445μs 86.7ns 324ns 37.9 2.23 0 239.88 KB
#6960 ObjectExtractorSimpleBody net6.0 317ns 1.71ns 8.87ns 0 0 0 280 B
#6960 ObjectExtractorSimpleBody netcoreapp3.1 410ns 0.824ns 3.19ns 0 0 0 272 B
#6960 ObjectExtractorSimpleBody net472 302ns 0.0184ns 0.0662ns 0.0441 0 0 281 B
#6960 ObjectExtractorMoreComplexBody net6.0 6.31μs 30.5ns 133ns 0 0 0 3.78 KB
#6960 ObjectExtractorMoreComplexBody netcoreapp3.1 7.78μs 21.3ns 82.4ns 0 0 0 3.69 KB
#6960 ObjectExtractorMoreComplexBody net472 6.69μs 6.28ns 24.3ns 0.602 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
master EncodeArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master EncodeArgs net472 N/A N/A N/A NaN NaN NaN 0 b
master EncodeLegacyArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
master EncodeLegacyArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master EncodeLegacyArgs net472 N/A N/A N/A NaN NaN NaN 0 b
#6960 EncodeArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
#6960 EncodeArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#6960 EncodeArgs net472 N/A N/A N/A NaN NaN NaN 0 b
#6960 EncodeLegacyArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
#6960 EncodeLegacyArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#6960 EncodeLegacyArgs net472 N/A N/A N/A NaN NaN NaN 0 b
Benchmarks.Trace.Asm.AppSecWafBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmark netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmark net472 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmarkWithAttack net6.0 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmarkWithAttack net472 N/A N/A N/A NaN NaN NaN 0 b
#6960 RunWafRealisticBenchmark net6.0 N/A N/A N/A NaN NaN NaN 0 b
#6960 RunWafRealisticBenchmark netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#6960 RunWafRealisticBenchmark net472 N/A N/A N/A NaN NaN NaN 0 b
#6960 RunWafRealisticBenchmarkWithAttack net6.0 N/A N/A N/A NaN NaN NaN 0 b
#6960 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#6960 RunWafRealisticBenchmarkWithAttack net472 N/A N/A N/A NaN NaN NaN 0 b
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 60.9μs 31.5ns 122ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 71.4μs 98.2ns 368ns 0 0 0 17.42 KB
master SendRequest net472 0.0191ns 0.00167ns 0.00647ns 0 0 0 0 b
#6960 SendRequest net6.0 61.9μs 31ns 112ns 0 0 0 14.53 KB
#6960 SendRequest netcoreapp3.1 71.1μs 226ns 845ns 0 0 0 17.42 KB
#6960 SendRequest net472 0.0254ns 0.0016ns 0.00619ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #6960

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.156 757,038.22 655,108.04 several?

More allocations ⚠️ in #6960

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.64 KB 42.43 KB 785 B 1.89%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 695μs 3.94μs 26.7μs 0 0 0 41.64 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 742μs 4.29μs 34.4μs 0 0 0 41.88 KB
master WriteAndFlushEnrichedTraces net472 972μs 3.16μs 12.2μs 4.46 0 0 55.85 KB
#6960 WriteAndFlushEnrichedTraces net6.0 755μs 2.54μs 9.83μs 0 0 0 42.43 KB
#6960 WriteAndFlushEnrichedTraces netcoreapp3.1 658μs 2.06μs 7.96μs 0 0 0 42.01 KB
#6960 WriteAndFlushEnrichedTraces net472 1.02ms 4.2μs 16.2μs 4.46 0 0 56.11 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.98μs 9.05ns 36.2ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.46μs 2.28ns 8.83ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.8μs 5.37ns 20.8ns 0.153 0.014 0 995 B
#6960 ExecuteNonQuery net6.0 1.84μs 9.29ns 42.6ns 0 0 0 1.03 KB
#6960 ExecuteNonQuery netcoreapp3.1 2.47μs 8.19ns 31.7ns 0 0 0 1.02 KB
#6960 ExecuteNonQuery net472 2.83μs 2.41ns 9.35ns 0.156 0.0142 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.75μs 7.98ns 31.9ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.33μs 9.99ns 38.7ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.46μs 2.47ns 9.57ns 0.155 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.85μs 1.36ns 4.89ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.33μs 10.7ns 41.5ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.77μs 4.59ns 17.8ns 0.169 0 0 1.11 KB
#6960 CallElasticsearch net6.0 1.79μs 1.72ns 6.2ns 0 0 0 1.04 KB
#6960 CallElasticsearch netcoreapp3.1 2.28μs 8.73ns 33.8ns 0 0 0 1.04 KB
#6960 CallElasticsearch net472 3.39μs 1.95ns 7.55ns 0.152 0 0 1.05 KB
#6960 CallElasticsearchAsync net6.0 1.86μs 9.59ns 45ns 0 0 0 1.02 KB
#6960 CallElasticsearchAsync netcoreapp3.1 2.53μs 5.38ns 20.9ns 0 0 0 1.09 KB
#6960 CallElasticsearchAsync net472 3.56μs 3.27ns 12.7ns 0.16 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.85μs 9.2ns 39ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.3μs 9.74ns 37.7ns 0 0 0 960 B
master ExecuteAsync net472 2.75μs 1.98ns 7.13ns 0.136 0 0 923 B
#6960 ExecuteAsync net6.0 1.76μs 7.1ns 27.5ns 0 0 0 960 B
#6960 ExecuteAsync netcoreapp3.1 2.31μs 7.52ns 29.1ns 0 0 0 960 B
#6960 ExecuteAsync net472 2.61μs 0.903ns 3.13ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.91μs 21.3ns 79.7ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.78μs 21ns 81.5ns 0 0 0 2.9 KB
master SendAsync net472 12.2μs 5.25ns 20.3ns 0.484 0 0 3.19 KB
#6960 SendAsync net6.0 6.88μs 23.6ns 91.6ns 0 0 0 2.37 KB
#6960 SendAsync netcoreapp3.1 8.34μs 8.22ns 31.8ns 0 0 0 2.9 KB
#6960 SendAsync net472 11.9μs 7.96ns 30.8ns 0.478 0 0 3.19 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #6960

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 2.775 1,385,600.00 499,400.00

More allocations ⚠️ in #6960

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 57.34 KB 65.54 KB 8.19 KB 14.29%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 43.78 KB 45.18 KB 1.4 KB 3.20%

Fewer allocations 🎉 in #6960

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 277.58 KB 275.26 KB -2.31 KB -0.83%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 342.2 KB 274.86 KB -67.34 KB -19.68%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 45.1μs 343ns 3.22μs 0 0 0 43.78 KB
master StringConcatBenchmark netcoreapp3.1 48.2μs 275ns 1.82μs 0 0 0 42.73 KB
master StringConcatBenchmark net472 57.1μs 200ns 749ns 0 0 0 57.34 KB
master StringConcatAspectBenchmark net6.0 1.39ms 2.27μs 7.87μs 0 0 0 342.2 KB
master StringConcatAspectBenchmark netcoreapp3.1 538μs 1.96μs 7.33μs 0 0 0 277.58 KB
master StringConcatAspectBenchmark net472 412μs 2.31μs 15.1μs 0 0 0 278.53 KB
#6960 StringConcatBenchmark net6.0 41.7μs 191ns 895ns 0 0 0 45.18 KB
#6960 StringConcatBenchmark netcoreapp3.1 46.8μs 200ns 799ns 0 0 0 42.7 KB
#6960 StringConcatBenchmark net472 58μs 248ns 961ns 0 0 0 65.54 KB
#6960 StringConcatAspectBenchmark net6.0 501μs 2.16μs 7.79μs 0 0 0 274.86 KB
#6960 StringConcatAspectBenchmark netcoreapp3.1 535μs 1.83μs 6.85μs 0 0 0 275.26 KB
#6960 StringConcatAspectBenchmark net472 408μs 2.25μs 13.5μs 0 0 0 278.53 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.61μs 4ns 15ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.47μs 11.7ns 45.3ns 0 0 0 1.76 KB
master EnrichedLog net472 3.93μs 2.14ns 7.7ns 0.255 0 0 1.69 KB
#6960 EnrichedLog net6.0 2.58μs 1.86ns 7.19ns 0 0 0 1.76 KB
#6960 EnrichedLog netcoreapp3.1 3.52μs 16.6ns 66.3ns 0 0 0 1.76 KB
#6960 EnrichedLog net472 3.87μs 5.31ns 20.6ns 0.252 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 122μs 44.1ns 159ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 127μs 443ns 1.6μs 0 0 0 4.37 KB
master EnrichedLog net472 167μs 96.1ns 360ns 0 0 0 4.57 KB
#6960 EnrichedLog net6.0 123μs 94.9ns 368ns 0 0 0 4.37 KB
#6960 EnrichedLog netcoreapp3.1 127μs 68.5ns 247ns 0 0 0 4.37 KB
#6960 EnrichedLog net472 168μs 150ns 563ns 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.07μs 20.6ns 80ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.84μs 5.2ns 19.4ns 0 0 0 2.32 KB
master EnrichedLog net472 7.29μs 3.56ns 13.3ns 0.326 0 0 2.14 KB
#6960 EnrichedLog net6.0 5.07μs 8.92ns 34.5ns 0 0 0 2.32 KB
#6960 EnrichedLog netcoreapp3.1 6.72μs 23ns 89.1ns 0 0 0 2.32 KB
#6960 EnrichedLog net472 7.41μs 11.2ns 43.5ns 0.33 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.06μs 2.43ns 9.11ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.55μs 11.9ns 47.6ns 0 0 0 1.21 KB
master SendReceive net472 3.05μs 2.5ns 9.69ns 0.183 0 0 1.21 KB
#6960 SendReceive net6.0 1.99μs 2.56ns 9.93ns 0 0 0 1.21 KB
#6960 SendReceive netcoreapp3.1 2.57μs 4.91ns 19ns 0 0 0 1.21 KB
#6960 SendReceive net472 3.05μs 7.09ns 27.5ns 0.183 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.47μs 3.87ns 14.5ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.76μs 19.1ns 74.1ns 0 0 0 1.69 KB
master EnrichedLog net472 6.87μs 8.56ns 33.2ns 0.312 0 0 2.08 KB
#6960 EnrichedLog net6.0 4.3μs 12.3ns 47.5ns 0 0 0 1.64 KB
#6960 EnrichedLog netcoreapp3.1 5.4μs 15.5ns 59.9ns 0 0 0 1.69 KB
#6960 EnrichedLog net472 6.43μs 5.12ns 19.8ns 0.32 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 745ns 3.85ns 14.9ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 946ns 4.5ns 17.4ns 0 0 0 584 B
master StartFinishSpan net472 919ns 0.235ns 0.88ns 0.0919 0 0 586 B
master StartFinishScope net6.0 923ns 0.334ns 1.29ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.16μs 5.44ns 21.8ns 0 0 0 704 B
master StartFinishScope net472 1.12μs 0.0531ns 0.184ns 0.101 0 0 666 B
#6960 StartFinishSpan net6.0 793ns 4.33ns 23.7ns 0 0 0 584 B
#6960 StartFinishSpan netcoreapp3.1 936ns 5.13ns 31.6ns 0 0 0 584 B
#6960 StartFinishSpan net472 888ns 0.202ns 0.783ns 0.0887 0 0 586 B
#6960 StartFinishScope net6.0 885ns 4.73ns 24.6ns 0 0 0 704 B
#6960 StartFinishScope netcoreapp3.1 1.16μs 5.63ns 23.9ns 0 0 0 704 B
#6960 StartFinishScope net472 1.1μs 0.364ns 1.36ns 0.106 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.03μs 0.448ns 1.74ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.37μs 2.42ns 8.73ns 0 0 0 704 B
master RunOnMethodBegin net472 1.38μs 1.11ns 4.3ns 0.104 0 0 666 B
#6960 RunOnMethodBegin net6.0 1.05μs 5.46ns 26.2ns 0 0 0 704 B
#6960 RunOnMethodBegin netcoreapp3.1 1.32μs 6.73ns 28.6ns 0 0 0 704 B
#6960 RunOnMethodBegin net472 1.34μs 1.77ns 6.84ns 0.101 0 0 666 B

NachoEchevarria
NachoEchevarria approved these changes May 12, 2025
View reviewed changes
Copy link
Contributor

@NachoEchevarria NachoEchevarria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@andrewlock
Copy link
Member Author

andrewlock commented May 12, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6960) - mean (68ms)  : 65, 70
     .   : milestone, 68,
    master - mean (68ms)  : 66, 69
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6960) - mean (1,000ms)  : 980, 1020
     .   : milestone, 1000,
    master - mean (1,004ms)  : 982, 1027
     .   : milestone, 1004,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6960) - mean (102ms)  : 100, 104
     .   : milestone, 102,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6960) - mean (689ms)  : 668, 711
     .   : milestone, 689,
    master - mean (694ms)  : 678, 710
     .   : milestone, 694,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6960) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (89ms)  : 86, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6960) - mean (650ms)  : 631, 669
     .   : milestone, 650,
    master - mean (653ms)  : 629, 678
     .   : milestone, 653,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6960) - mean (189ms)  : 185, 194
     .   : milestone, 189,
    master - mean (189ms)  : 185, 193
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (6960) - mean (1,112ms)  : 1085, 1138
     .   : milestone, 1112,
    master - mean (1,114ms)  : 1086, 1142
     .   : milestone, 1114,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6960) - mean (268ms)  : 264, 273
     .   : milestone, 268,
    master - mean (269ms)  : 265, 272
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (6960) - mean (877ms)  : 847, 908
     .   : milestone, 877,
    master - mean (884ms)  : 851, 917
     .   : milestone, 884,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6960) - mean (261ms)  : 258, 264
     .   : milestone, 261,
    master - mean (261ms)  : 257, 265
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (6960) - mean (867ms)  : 830, 903
     .   : milestone, 867,
    master - mean (869ms)  : 836, 902
     .   : milestone, 869,
Loading

@andrewlock andrewlock force-pushed the andrew/ci/switch-to-managed-devops-pools branch from 6731f3a to 674406a Compare May 13, 2025 09:47
@andrewlock andrewlock requested a review from a team as a code owner May 14, 2025 12:17
@andrewlock andrewlock force-pushed the andrew/ci/switch-to-managed-devops-pools branch from 71325d7 to 248d989 Compare May 14, 2025 12:28
@andrewlock andrewlock force-pushed the andrew/ci/switch-to-managed-devops-pools branch from 248d989 to 1cf846b Compare May 14, 2025 12:33
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented May 14, 2025 •
edited
Loading

Datadog Report

All test runs 1cf846b 🔗

✅ 2 Total Test Services: 0 Failed, 2 Passed

Test Services
Service Name Failed Known Flaky New Flaky Passed Skipped Total Time Test Service View
dd-trace-dotnet 0 0 0 250907 2308 17h 49m 47.83s Link
exploration_tests 0 0 0 22085 3 2m 17.12s Link

andrewlock
andrewlock commented May 14, 2025
View reviewed changes
.azure-pipelines/ultimate-pipeline.yml
@@ -135,7 +135,6 @@ variables:
DOTNET_CLI_UI_LANGUAGE: 'en'
DOTNET_MULTILEVEL_LOOKUP: 0
NUGET_XMLDOC_MODE: 'skip'
DOTNET_GENERATE_ASPNET_CERTIFICATE: false
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to remove this because we were getting crashes in the .NET 8 and .NET 9 on Windows grpc tests 😅

image

andrewlock
andrewlock commented May 14, 2025
View reviewed changes
tracer/test/Datadog.Trace.TestHelpers.SharedSource/VerifyHelper.cs
path switch
{
_ when path.StartsWith(@"D:/a/1/s/") => path.Substring(9),
_ when path.StartsWith(@"D:/a/_work/1/s/") => path.Substring(15),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new hosted pools use a slightly different checkout path, which broke the file scrubbing for CI vis tests

@andrewlock andrewlock merged commit 3fe1f6f into master May 14, 2025
129 checks passed
@andrewlock andrewlock deleted the andrew/ci/switch-to-managed-devops-pools branch May 14, 2025 14:45
@github-actions github-actions bot added this to the vNext-v3 milestone May 14, 2025
chojomok pushed a commit that referenced this pull request Jul 15, 2025
@andrewlock @chojomok
Update Windows to hosted managed DevOps pool (#6960)
05da983 
## Summary of changes

Move the Windows VMs from a VMSS pool to use a Hosted Managed DevOps
Pool

## Reason for change

We want to unify the infrastructure hence the move from VMSS to hosted
pools

## Implementation details

- Set up a new hosted devops pool (I will document this process in
confluence once I'm/we're happy)
- Reused the same image, so should have identical behaviour otherwise
- Update ultimate-pipeline to use the new pool

## Test coverage

This is the test

## Other details

Supersedes
- #6819

As it's a new pool, we have to _manually_ specify pre-provisioning. I've
currently set this to 9-5 UTC with 10VMs but we can tweak that as we see
fit. It's currently just to ensure we're getting some pre-provisioning
benefit. We can tweak that as we see fit (and can use predictive mode in
~3 weeks time, once Azure has time to analyze our usage)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bouwkast bouwkast bouwkast approved these changes

@NachoEchevarria NachoEchevarria NachoEchevarria approved these changes

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.17.0
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewlock @bouwkast @NachoEchevarria