Skip to content

Add documentation around Xunit.Combinatorial #6880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Apr 23, 2025
Merged

Add documentation around Xunit.Combinatorial #6880

merged 4 commits into from
Apr 23, 2025

Conversation

bouwkast
Copy link
Collaborator

Summary of changes

Adds a basic development doc around Xunit.Combinatorial.

Reason for change

Attempt to give it an overview and how to migrate current tests to it and best practices.

Implementation details

Test coverage

Other details

@bouwkast bouwkast requested a review from a team as a code owner April 22, 2025 18:45
@bouwkast bouwkast requested a review from Copilot April 22, 2025 18:45
Copilot
Copilot AI reviewed Apr 22, 2025
View reviewed changes
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a development document to help teams understand and migrate tests to use Xunit.Combinatorial, highlighting its benefits and best practices.

  • Introduces an overview of Xunit.Combinatorial including performance gains
  • Describes migration steps, configuration options, and best practices for its usage

@pr-commenter
Copy link

pr-commenter bot commented Apr 22, 2025 •
edited
Loading

Benchmarks

Benchmarks Report for tracer 🐌

Benchmarks for #6880 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.147
  • 1 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.2μs 54.9ns 320ns 0 0 0 5.55 KB
master StartStopWithChild netcoreapp3.1 13.7μs 40.3ns 156ns 0 0 0 5.77 KB
master StartStopWithChild net472 22.2μs 113ns 518ns 0.909 0.227 0 6.18 KB
#6880 StartStopWithChild net6.0 10.1μs 56.4ns 378ns 0 0 0 5.54 KB
#6880 StartStopWithChild netcoreapp3.1 14.3μs 70.6ns 300ns 0 0 0 5.77 KB
#6880 StartStopWithChild net472 21.5μs 116ns 668ns 0.994 0.331 0.11 6.17 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 934μs 178ns 668ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 178ns 666ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.22ms 237ns 888ns 0 0 0 3.31 KB
#6880 WriteAndFlushEnrichedTraces net6.0 932μs 178ns 643ns 0 0 0 2.71 KB
#6880 WriteAndFlushEnrichedTraces netcoreapp3.1 1.04ms 96.8ns 349ns 0 0 0 2.7 KB
#6880 WriteAndFlushEnrichedTraces net472 1.22ms 409ns 1.58μs 0 0 0 3.31 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61.3μs 44.8ns 155ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 70.9μs 60.3ns 218ns 0 0 0 17.42 KB
master SendRequest net472 0.00947ns 0.00264ns 0.0102ns 0 0 0 0 b
#6880 SendRequest net6.0 60μs 79.4ns 297ns 0 0 0 14.53 KB
#6880 SendRequest netcoreapp3.1 70.8μs 68.4ns 256ns 0 0 0 17.42 KB
#6880 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #6880

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.147 713,959.86 622,200.00

More allocations ⚠️ in #6880

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 55.95 KB 56.29 KB 340 B 0.61%

Fewer allocations 🎉 in #6880

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 42.69 KB 41.94 KB -751 B -1.76%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 729μs 4.16μs 30μs 0 0 0 41.77 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 714μs 688ns 2.57μs 0 0 0 42.69 KB
master WriteAndFlushEnrichedTraces net472 899μs 3.38μs 13.1μs 8.33 0 0 55.95 KB
#6880 WriteAndFlushEnrichedTraces net6.0 778μs 1.57μs 6.1μs 0 0 0 41.79 KB
#6880 WriteAndFlushEnrichedTraces netcoreapp3.1 627μs 2.37μs 13.2μs 0 0 0 41.94 KB
#6880 WriteAndFlushEnrichedTraces net472 977μs 2.29μs 8.56μs 8.93 4.46 0 56.29 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.95μs 8.15ns 31.6ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.49μs 1.92ns 7.46ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.78μs 2.5ns 9.68ns 0.152 0.0138 0 995 B
#6880 ExecuteNonQuery net6.0 1.94μs 7.88ns 30.5ns 0 0 0 1.03 KB
#6880 ExecuteNonQuery netcoreapp3.1 2.48μs 12.6ns 56.4ns 0 0 0 1.02 KB
#6880 ExecuteNonQuery net472 2.8μs 2.54ns 9.49ns 0.152 0.0138 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.81μs 0.673ns 2.61ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.4μs 2.9ns 11.2ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.52μs 2.48ns 9.29ns 0.159 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.87μs 8.61ns 33.3ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.43μs 8.99ns 34.8ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.6μs 1.16ns 4.48ns 0.162 0 0 1.11 KB
#6880 CallElasticsearch net6.0 1.82μs 9.03ns 38.3ns 0 0 0 1.04 KB
#6880 CallElasticsearch netcoreapp3.1 2.31μs 10.8ns 41.9ns 0 0 0 1.04 KB
#6880 CallElasticsearch net472 3.62μs 1.48ns 5.54ns 0.163 0 0 1.05 KB
#6880 CallElasticsearchAsync net6.0 1.88μs 8.37ns 32.4ns 0 0 0 1.02 KB
#6880 CallElasticsearchAsync netcoreapp3.1 2.45μs 8.02ns 31.1ns 0 0 0 1.09 KB
#6880 CallElasticsearchAsync net472 3.74μs 5.96ns 23.1ns 0.168 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.78μs 8.32ns 34.3ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.29μs 9.85ns 38.2ns 0 0 0 960 B
master ExecuteAsync net472 2.55μs 4.77ns 18.5ns 0.14 0 0 923 B
#6880 ExecuteAsync net6.0 1.83μs 2.46ns 8.86ns 0 0 0 960 B
#6880 ExecuteAsync netcoreapp3.1 2.43μs 2.27ns 8.49ns 0 0 0 960 B
#6880 ExecuteAsync net472 2.67μs 0.487ns 1.89ns 0.146 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 7.21μs 21.4ns 80ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.33μs 11.7ns 45.4ns 0 0 0 2.9 KB
master SendAsync net472 12.6μs 11.4ns 42.6ns 0.502 0 0 3.19 KB
#6880 SendAsync net6.0 6.76μs 8.04ns 30.1ns 0 0 0 2.37 KB
#6880 SendAsync netcoreapp3.1 8.43μs 31.9ns 124ns 0 0 0 2.9 KB
#6880 SendAsync net472 12.6μs 10.8ns 42ns 0.506 0 0 3.19 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.71μs 11.6ns 44.9ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.37μs 17.4ns 83.7ns 0 0 0 1.76 KB
master EnrichedLog net472 4.02μs 3.64ns 14.1ns 0.26 0 0 1.69 KB
#6880 EnrichedLog net6.0 2.63μs 12.6ns 53.4ns 0 0 0 1.76 KB
#6880 EnrichedLog netcoreapp3.1 3.46μs 2.02ns 7.81ns 0 0 0 1.76 KB
#6880 EnrichedLog net472 3.97μs 5.09ns 19ns 0.258 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 124μs 34.3ns 124ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 128μs 111ns 401ns 0 0 0 4.37 KB
master EnrichedLog net472 167μs 28.5ns 110ns 0 0 0 4.57 KB
#6880 EnrichedLog net6.0 123μs 75.9ns 263ns 0 0 0 4.37 KB
#6880 EnrichedLog netcoreapp3.1 129μs 305ns 1.14μs 0 0 0 4.37 KB
#6880 EnrichedLog net472 168μs 78ns 302ns 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.12μs 20.8ns 77.7ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.7μs 10.7ns 41.4ns 0 0 0 2.32 KB
master EnrichedLog net472 7.29μs 5.38ns 20.8ns 0.327 0 0 2.14 KB
#6880 EnrichedLog net6.0 5.11μs 8.84ns 34.2ns 0 0 0 2.32 KB
#6880 EnrichedLog netcoreapp3.1 6.71μs 17.3ns 64.6ns 0 0 0 2.32 KB
#6880 EnrichedLog net472 7.46μs 8.98ns 33.6ns 0.335 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.98μs 10.2ns 48ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.63μs 10.6ns 41.1ns 0 0 0 1.21 KB
master SendReceive net472 3.2μs 9.37ns 36.3ns 0.191 0 0 1.21 KB
#6880 SendReceive net6.0 2.05μs 9.54ns 37ns 0 0 0 1.21 KB
#6880 SendReceive netcoreapp3.1 2.59μs 12.6ns 54.9ns 0 0 0 1.21 KB
#6880 SendReceive net472 3.16μs 1.1ns 4.11ns 0.19 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.14μs 17.4ns 67.4ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.84μs 12.3ns 47.6ns 0 0 0 1.69 KB
master EnrichedLog net472 6.74μs 8ns 31ns 0.301 0 0 2.08 KB
#6880 EnrichedLog net6.0 4.28μs 4.81ns 18.6ns 0 0 0 1.64 KB
#6880 EnrichedLog netcoreapp3.1 5.71μs 18.4ns 71.3ns 0 0 0 1.69 KB
#6880 EnrichedLog net472 6.75μs 5.88ns 22.8ns 0.304 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 745ns 3.64ns 15ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 938ns 5.04ns 26.2ns 0 0 0 584 B
master StartFinishSpan net472 922ns 0.298ns 1.07ns 0.0923 0 0 586 B
master StartFinishScope net6.0 908ns 0.612ns 2.37ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.17μs 2.63ns 10.2ns 0 0 0 704 B
master StartFinishScope net472 1.1μs 1.85ns 7.18ns 0.104 0 0 666 B
#6880 StartFinishSpan net6.0 757ns 0.328ns 1.27ns 0 0 0 584 B
#6880 StartFinishSpan netcoreapp3.1 934ns 5.17ns 28.8ns 0 0 0 584 B
#6880 StartFinishSpan net472 914ns 0.25ns 0.967ns 0.0919 0 0 586 B
#6880 StartFinishScope net6.0 898ns 2.81ns 10.5ns 0 0 0 704 B
#6880 StartFinishScope netcoreapp3.1 1.17μs 5.58ns 21.6ns 0 0 0 704 B
#6880 StartFinishScope net472 1.1μs 0.0827ns 0.31ns 0.105 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.03μs 5.03ns 21.9ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.39μs 0.651ns 2.44ns 0 0 0 704 B
master RunOnMethodBegin net472 1.4μs 0.965ns 3.74ns 0.104 0 0 666 B
#6880 RunOnMethodBegin net6.0 1.03μs 1.32ns 5.11ns 0 0 0 704 B
#6880 RunOnMethodBegin netcoreapp3.1 1.34μs 5.6ns 21.7ns 0 0 0 704 B
#6880 RunOnMethodBegin net472 1.38μs 0.628ns 2.35ns 0.104 0 0 666 B

andrewlock
andrewlock approved these changes Apr 23, 2025
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks for doing this!

@bouwkast @andrewlock
Add Andrew's changes
41075d6 
Co-authored-by: Andrew Lock <[email protected]>
@bouwkast bouwkast merged commit 60fd0de into master Apr 23, 2025
99 of 102 checks passed
@bouwkast bouwkast deleted the steven/combinatorial-docs branch April 23, 2025 18:37
@github-actions github-actions bot added this to the vNext-v3 milestone Apr 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:docs
Projects
None yet
Milestone
3.16.0
Development

Successfully merging this pull request may close these issues.
2 participants
@bouwkast @andrewlock