Skip to content

[IAST] LDAP flakiness fix #6706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 11, 2025
Merged

[IAST] LDAP flakiness fix #6706

merged 3 commits into from
Mar 11, 2025

Conversation

daniel-romano-DD
Copy link
Contributor

@daniel-romano-DD daniel-romano-DD commented Feb 24, 2025 •
edited
Loading

Summary of changes

Handle server behavior when it's not ready to handle the LDAP request. Slip test in this case

Reason for change

Sometimes LDAP requests crash in CI, unrelated to our instrumentation

Implementation details

Moved LDAP requesto to a task and handle the timeout and possible COMException thrown

Test coverage

Other details

@daniel-romano-DD daniel-romano-DD added area:tests unit tests, integration tests area:asm-iast labels Feb 24, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 24, 2025 •
edited
Loading

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

3 occurrences of :

-      http.url: http://localhost:00000/Iast/Ldap?userName=Babs%20Jensen,
+      http.url: http://localhost:00000/Iast/Ldap?path=LDAP://ldap.forumsys.com:389/dc=example,dc=com,

1 occurrences of :

-      "hash": 941004385,
+      "hash": 9515978
[...]
-        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
-        "method": "Ldap"
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController+"
+        "method": "<Ldap>g__PerformLdapQuery|0"
[...]
-            "value": "(uid="
-          },
-          {
-            "redacted": true,
-            "pattern": "abcdefghijk",
+            "value": "LDAP://ldap.forumsys.com:389/dc=example,dc=com",
[...]
-          },
-          {
-            "value": ")"

1 occurrences of :

-      "name": "userName",
-      "redacted": true,
-      "pattern": "abcdefghijk"
+      "name": "path",
+      "value": "LDAP://ldap.forumsys.com:389/dc=example,dc=com"

1 occurrences of :

-      "hash": 941004385,
+      "hash": 9515978
[...]
-        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
-        "method": "Ldap"
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController+"
+        "method": "<Ldap>g__PerformLdapQuery|0"
[...]
-            "value": "(uid="
-          },
-          {
-            "value": "Babs Jensen",
+            "value": "LDAP://ldap.forumsys.com:389/dc=example,dc=com",
[...]
-          },
-          {
-            "value": ")"

1 occurrences of :

-      "name": "userName",
-      "value": "Babs Jensen"
+      "name": "path",
+      "value": "LDAP://ldap.forumsys.com:389/dc=example,dc=com"

4 occurrences of :

-      http.url: http://localhost:00000/Iast/Ldap?path=LDAP://fakeorg,DC=com&userName=BabsJensen,
+      http.url: http://localhost:00000/Iast/Ldap?path=LDAP://ldap.forumsys.com:389/dc=example,dc=com,

1 occurrences of :

-      "hash": 941004385,
+      "hash": 9515978,
[...]
-        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
-        "method": "Ldap"
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController+<>c__DisplayClass28_0",
+        "method": "<Ldap>g__PerformLdapQuery|0"
[...]
-            "value": "LDAP://fakeorg,DC=com",
+            "value": "LDAP://ldap.forumsys.com:389/dc=example,dc=com",
[...]
-    },
-    {
-      "type": "LDAP_INJECTION",
-      "hash": 941004385,
-      "location": {
-        "spanId": XXX,
-        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
-        "method": "Ldap"
-      },
-      "evidence": {
-        "valueParts": [
-          {
-            "value": "(uid="
-          },
-          {
-            "value": "BabsJensen",
-            "source": 1
-          },
-          {
-            "value": ")"
-          }
-        ]
-      }
[...]
-      "value": "LDAP://fakeorg,DC=com"
-    },
-    {
-      "origin": "http.request.parameter",
-      "name": "userName",
-      "value": "BabsJensen"
+      "value": "LDAP://ldap.forumsys.com:389/dc=example,dc=com"

@andrewlock
Copy link
Member

andrewlock commented Feb 24, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6706) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6706) - mean (1,002ms)  : 975, 1029
     .   : milestone, 1002,
    master - mean (1,005ms)  : 981, 1029
     .   : milestone, 1005,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6706) - mean (103ms)  : 101, 105
     .   : milestone, 103,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6706) - mean (684ms)  : 667, 701
     .   : milestone, 684,
    master - mean (687ms)  : 671, 704
     .   : milestone, 687,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6706) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6706) - mean (639ms)  : 621, 658
     .   : milestone, 639,
    master - mean (646ms)  : 627, 665
     .   : milestone, 646,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6706) - mean (191ms)  : 187, 194
     .   : milestone, 191,
    master - mean (191ms)  : 187, 194
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6706) - mean (1,109ms)  : 1076, 1142
     .   : milestone, 1109,
    master - mean (1,110ms)  : 1082, 1138
     .   : milestone, 1110,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6706) - mean (269ms)  : 264, 273
     .   : milestone, 269,
    master - mean (270ms)  : 266, 273
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (6706) - mean (873ms)  : 843, 903
     .   : milestone, 873,
    master - mean (880ms)  : 846, 914
     .   : milestone, 880,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6706) - mean (261ms)  : 256, 266
     .   : milestone, 261,
    master - mean (262ms)  : 259, 265
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6706) - mean (851ms)  : 814, 888
     .   : milestone, 851,
    master - mean (856ms)  : 825, 886
     .   : milestone, 856,
Loading

@andrewlock
Copy link
Member

andrewlock commented Feb 24, 2025 •
edited
Loading

Benchmarks Report for tracer 🐌

Benchmarks for #6706 compared to master:

  • All benchmarks have the same speed
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.41μs 48.1ns 360ns 0.0166 0.00828 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10.3μs 56.7ns 345ns 0.0203 0.00509 0 5.81 KB
master StartStopWithChild net472 16.3μs 37.2ns 144ns 1.06 0.323 0.103 6.22 KB
#6706 StartStopWithChild net6.0 8.32μs 47.2ns 320ns 0.0195 0.00782 0 5.61 KB
#6706 StartStopWithChild netcoreapp3.1 10.6μs 61.2ns 501ns 0.0169 0.00564 0 5.8 KB
#6706 StartStopWithChild net472 16μs 34.5ns 134ns 1.06 0.333 0.095 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 502μs 445ns 1.67μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 637μs 213ns 797ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 858μs 441ns 1.65μs 0.428 0 0 3.3 KB
#6706 WriteAndFlushEnrichedTraces net6.0 501μs 279ns 1.08μs 0 0 0 2.7 KB
#6706 WriteAndFlushEnrichedTraces netcoreapp3.1 670μs 905ns 3.5μs 0 0 0 2.7 KB
#6706 WriteAndFlushEnrichedTraces net472 858μs 665ns 2.57μs 0.425 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 131μs 377ns 1.46μs 0.193 0 0 14.47 KB
master SendRequest netcoreapp3.1 153μs 135ns 523ns 0.231 0 0 17.27 KB
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#6706 SendRequest net6.0 135μs 526ns 2.04μs 0.197 0 0 14.47 KB
#6706 SendRequest netcoreapp3.1 150μs 277ns 1.07μs 0.224 0 0 17.27 KB
#6706 SendRequest net472 0.000638ns 0.000301ns 0.00113ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6706

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.42 KB 41.74 KB 316 B 0.76%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 549μs 2.29μs 8.27μs 0.566 0 0 41.42 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 690μs 3.89μs 26.7μs 0.355 0 0 41.77 KB
master WriteAndFlushEnrichedTraces net472 875μs 4.58μs 22.9μs 8.08 2.55 0.425 53.3 KB
#6706 WriteAndFlushEnrichedTraces net6.0 592μs 3.37μs 26.3μs 0.539 0 0 41.74 KB
#6706 WriteAndFlushEnrichedTraces netcoreapp3.1 724μs 4.46μs 44.6μs 0.347 0 0 41.79 KB
#6706 WriteAndFlushEnrichedTraces net472 871μs 4.35μs 19μs 8.08 2.55 0.425 53.29 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.34μs 1.62ns 6.28ns 0.0141 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.74μs 1.07ns 4.02ns 0.014 0 0 1.02 KB
master ExecuteNonQuery net472 2.06μs 1.72ns 6.42ns 0.157 0.00103 0 987 B
#6706 ExecuteNonQuery net6.0 1.29μs 0.618ns 2.31ns 0.0144 0 0 1.02 KB
#6706 ExecuteNonQuery netcoreapp3.1 1.81μs 2.11ns 8.19ns 0.0135 0 0 1.02 KB
#6706 ExecuteNonQuery net472 2.01μs 1.87ns 7ns 0.156 0.001 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.25μs 0.606ns 2.35ns 0.0138 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.56μs 0.955ns 3.7ns 0.0134 0 0 976 B
master CallElasticsearch net472 2.4μs 1.82ns 6.56ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.37μs 0.515ns 1.99ns 0.013 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.79ns 3.06ns 0.0139 0 0 1.02 KB
master CallElasticsearchAsync net472 2.63μs 1.38ns 4.97ns 0.166 0 0 1.05 KB
#6706 CallElasticsearch net6.0 1.26μs 0.631ns 2.36ns 0.0139 0 0 976 B
#6706 CallElasticsearch netcoreapp3.1 1.52μs 0.648ns 2.42ns 0.0129 0 0 976 B
#6706 CallElasticsearch net472 2.6μs 1.28ns 4.97ns 0.157 0 0 995 B
#6706 CallElasticsearchAsync net6.0 1.28μs 0.678ns 2.54ns 0.0134 0 0 952 B
#6706 CallElasticsearchAsync netcoreapp3.1 1.69μs 0.434ns 1.5ns 0.0135 0 0 1.02 KB
#6706 CallElasticsearchAsync net472 2.55μs 1.87ns 7.24ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.26μs 0.49ns 1.83ns 0.0133 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.58μs 0.956ns 3.58ns 0.0133 0 0 952 B
master ExecuteAsync net472 1.86μs 0.395ns 1.48ns 0.145 0 0 915 B
#6706 ExecuteAsync net6.0 1.31μs 0.488ns 1.69ns 0.0131 0 0 952 B
#6706 ExecuteAsync netcoreapp3.1 1.65μs 0.398ns 1.44ns 0.0126 0 0 952 B
#6706 ExecuteAsync net472 1.89μs 0.807ns 3.02ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.4μs 1.16ns 4.33ns 0.0308 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.4μs 1.66ns 6.44ns 0.0377 0 0 2.85 KB
master SendAsync net472 7.4μs 0.882ns 3.3ns 0.493 0 0 3.12 KB
#6706 SendAsync net6.0 4.48μs 3.03ns 11.7ns 0.0312 0 0 2.31 KB
#6706 SendAsync netcoreapp3.1 5.35μs 2.85ns 11ns 0.0376 0 0 2.85 KB
#6706 SendAsync net472 7.43μs 1.89ns 7.33ns 0.494 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.52μs 7.33ns 28.4ns 0.0231 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.34μs 0.862ns 3.34ns 0.0221 0 0 1.64 KB
master EnrichedLog net472 2.72μs 1.64ns 6.36ns 0.249 0 0 1.57 KB
#6706 EnrichedLog net6.0 1.5μs 1.59ns 6.17ns 0.0226 0 0 1.64 KB
#6706 EnrichedLog netcoreapp3.1 2.23μs 1.61ns 6.22ns 0.0223 0 0 1.64 KB
#6706 EnrichedLog net472 2.65μs 0.916ns 3.43ns 0.248 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 179ns 693ns 0.056 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 119μs 186ns 722ns 0.0591 0 0 4.28 KB
master EnrichedLog net472 149μs 159ns 594ns 0.667 0.222 0 4.46 KB
#6706 EnrichedLog net6.0 113μs 186ns 721ns 0.0568 0 0 4.28 KB
#6706 EnrichedLog netcoreapp3.1 117μs 199ns 769ns 0.0582 0 0 4.28 KB
#6706 EnrichedLog net472 150μs 213ns 825ns 0.671 0.224 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.08μs 0.908ns 3.4ns 0.0308 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.48μs 1.19ns 4.47ns 0.0291 0 0 2.2 KB
master EnrichedLog net472 5.05μs 1.08ns 4.03ns 0.32 0 0 2.02 KB
#6706 EnrichedLog net6.0 3.16μs 0.871ns 3.37ns 0.0315 0 0 2.2 KB
#6706 EnrichedLog netcoreapp3.1 4.07μs 1.29ns 4.83ns 0.0304 0 0 2.2 KB
#6706 EnrichedLog net472 4.86μs 1.23ns 4.76ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.37μs 0.643ns 2.41ns 0.0164 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.8μs 0.718ns 2.78ns 0.0153 0 0 1.14 KB
master SendReceive net472 2.12μs 1.48ns 5.72ns 0.184 0 0 1.16 KB
#6706 SendReceive net6.0 1.4μs 0.757ns 2.93ns 0.0161 0 0 1.14 KB
#6706 SendReceive netcoreapp3.1 1.76μs 1.18ns 4.58ns 0.0149 0 0 1.14 KB
#6706 SendReceive net472 2.08μs 0.766ns 2.76ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.76μs 7.49ns 29ns 0.0221 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 4.15μs 2.95ns 11.4ns 0.0207 0 0 1.65 KB
master EnrichedLog net472 4.19μs 1.63ns 5.65ns 0.322 0 0 2.04 KB
#6706 EnrichedLog net6.0 2.82μs 1.28ns 4.77ns 0.0226 0 0 1.6 KB
#6706 EnrichedLog netcoreapp3.1 4.07μs 4.05ns 15.7ns 0.0225 0 0 1.65 KB
#6706 EnrichedLog net472 4.29μs 1.91ns 6.89ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 396ns 0.703ns 2.72ns 0.00809 0 0 576 B
master StartFinishSpan netcoreapp3.1 586ns 1.59ns 6.15ns 0.00764 0 0 576 B
master StartFinishSpan net472 638ns 1.23ns 4.75ns 0.0918 0 0 578 B
master StartFinishScope net6.0 485ns 0.955ns 3.7ns 0.00989 0 0 696 B
master StartFinishScope netcoreapp3.1 731ns 1.44ns 5.58ns 0.00923 0 0 696 B
master StartFinishScope net472 830ns 1.71ns 6.41ns 0.105 0 0 658 B
#6706 StartFinishSpan net6.0 388ns 0.444ns 1.72ns 0.00815 0 0 576 B
#6706 StartFinishSpan netcoreapp3.1 543ns 0.856ns 3.31ns 0.0077 0 0 576 B
#6706 StartFinishSpan net472 670ns 1.54ns 5.97ns 0.0916 0 0 578 B
#6706 StartFinishScope net6.0 533ns 0.75ns 2.9ns 0.00969 0 0 696 B
#6706 StartFinishScope netcoreapp3.1 704ns 1.16ns 4.48ns 0.00929 0 0 696 B
#6706 StartFinishScope net472 790ns 2.19ns 8.49ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 649ns 1.38ns 5.35ns 0.00969 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 913ns 1.64ns 6.35ns 0.00942 0 0 696 B
master RunOnMethodBegin net472 1.04μs 2.22ns 8.6ns 0.104 0 0 658 B
#6706 RunOnMethodBegin net6.0 658ns 0.615ns 2.38ns 0.00992 0 0 696 B
#6706 RunOnMethodBegin netcoreapp3.1 912ns 0.817ns 3.06ns 0.00961 0 0 696 B
#6706 RunOnMethodBegin net472 1.01μs 1.28ns 4.97ns 0.104 0 0 658 B

@andrewlock
Copy link
Member

andrewlock commented Feb 24, 2025 •
edited
Loading

Benchmarks Report for appsec 🐌

Benchmarks for #6706 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.148
  • 1 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6706

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0 1.117 195,611.41 218,498.99

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 191μs 33ns 119ns 2.77 0 0 195.5 KB
master AllCycleSimpleBody netcoreapp3.1 293μs 103ns 386ns 2.77 0 0 203.03 KB
master AllCycleSimpleBody net472 257μs 116ns 436ns 37.2 2.19 0 233.84 KB
master AllCycleMoreComplexBody net6.0 196μs 66.4ns 248ns 2.75 0 0 199.01 KB
master AllCycleMoreComplexBody netcoreapp3.1 305μs 103ns 400ns 2.74 0 0 206.44 KB
master AllCycleMoreComplexBody net472 264μs 89.2ns 334ns 37.7 2.11 0 237.35 KB
master ObjectExtractorSimpleBody net6.0 137ns 0.152ns 0.57ns 0.00396 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 191ns 0.0675ns 0.234ns 0.00374 0 0 272 B
master ObjectExtractorSimpleBody net472 211ns 0.151ns 0.585ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.96μs 1.99ns 7.46ns 0.053 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.83μs 3.21ns 12.4ns 0.05 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 4.42μs 2.52ns 9.43ns 0.603 0.00662 0 3.8 KB
#6706 AllCycleSimpleBody net6.0 209μs 102ns 380ns 2.74 0 0 195.48 KB
#6706 AllCycleSimpleBody netcoreapp3.1 316μs 100ns 361ns 2.68 0 0 203 KB
#6706 AllCycleSimpleBody net472 278μs 146ns 545ns 37.1 2.21 0 233.81 KB
#6706 AllCycleMoreComplexBody net6.0 219μs 99.5ns 372ns 2.88 0 0 198.98 KB
#6706 AllCycleMoreComplexBody netcoreapp3.1 312μs 127ns 474ns 2.8 0 0 206.42 KB
#6706 AllCycleMoreComplexBody net472 286μs 834ns 3.12μs 37.6 2.12 0 237.32 KB
#6706 ObjectExtractorSimpleBody net6.0 137ns 0.182ns 0.704ns 0.00396 0 0 280 B
#6706 ObjectExtractorSimpleBody netcoreapp3.1 206ns 0.193ns 0.723ns 0.00372 0 0 272 B
#6706 ObjectExtractorSimpleBody net472 210ns 0.235ns 0.91ns 0.0446 0 0 281 B
#6706 ObjectExtractorMoreComplexBody net6.0 3.14μs 1.36ns 5.11ns 0.0528 0 0 3.78 KB
#6706 ObjectExtractorMoreComplexBody netcoreapp3.1 3.8μs 2.46ns 9.54ns 0.0512 0 0 3.69 KB
#6706 ObjectExtractorMoreComplexBody net472 4.45μs 3.48ns 13.5ns 0.602 0.00672 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 38.3μs 32.3ns 125ns 0.457 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.8μs 238ns 1.06μs 0.431 0 0 32.4 KB
master EncodeArgs net472 66.1μs 38.1ns 148ns 5.15 0.066 0 32.5 KB
master EncodeLegacyArgs net6.0 72.9μs 43.3ns 150ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 108μs 48.5ns 181ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 151μs 65.7ns 255ns 0.305 0 0 2.15 KB
#6706 EncodeArgs net6.0 37.2μs 32.6ns 126ns 0.448 0 0 32.4 KB
#6706 EncodeArgs netcoreapp3.1 54μs 20.4ns 78.9ns 0.43 0 0 32.4 KB
#6706 EncodeArgs net472 66.1μs 43.6ns 163ns 5.14 0.0663 0 32.5 KB
#6706 EncodeLegacyArgs net6.0 78.8μs 33ns 128ns 0.0366 0 0 2.14 KB
#6706 EncodeLegacyArgs netcoreapp3.1 107μs 417ns 1.61μs 0 0 0 2.14 KB
#6706 EncodeLegacyArgs net472 151μs 94.3ns 365ns 0.301 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 174μs 103ns 387ns 0 0 0 2.54 KB
master RunWafRealisticBenchmark netcoreapp3.1 188μs 364ns 1.41μs 0 0 0 2.49 KB
master RunWafRealisticBenchmark net472 199μs 64.5ns 250ns 0.394 0 0 2.55 KB
master RunWafRealisticBenchmarkWithAttack net6.0 117μs 108ns 420ns 0 0 0 1.57 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 124μs 109ns 409ns 0 0 0 1.55 KB
master RunWafRealisticBenchmarkWithAttack net472 133μs 105ns 407ns 0.199 0 0 1.58 KB
#6706 RunWafRealisticBenchmark net6.0 173μs 111ns 431ns 0 0 0 2.54 KB
#6706 RunWafRealisticBenchmark netcoreapp3.1 190μs 138ns 535ns 0 0 0 2.49 KB
#6706 RunWafRealisticBenchmark net472 198μs 58.8ns 220ns 0.396 0 0 2.55 KB
#6706 RunWafRealisticBenchmarkWithAttack net6.0 116μs 88.8ns 344ns 0 0 0 1.57 KB
#6706 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 123μs 37.6ns 135ns 0 0 0 1.55 KB
#6706 RunWafRealisticBenchmarkWithAttack net472 133μs 39.6ns 154ns 0.199 0 0 1.58 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #6706

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1 1.180 53,900.00 63,600.00 bimodal

More allocations ⚠️ in #6706

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 57.64 KB 59.83 KB 2.19 KB 3.80%

Fewer allocations 🎉 in #6706

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 254.91 KB 252.89 KB -2.02 KB -0.79%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 64.3μs 836ns 8.32μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 53.9μs 202ns 728ns 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.5μs 105ns 435ns 0 0 0 57.64 KB
master StringConcatAspectBenchmark net6.0 306μs 1.66μs 9.25μs 0 0 0 253.83 KB
master StringConcatAspectBenchmark netcoreapp3.1 331μs 5.89μs 58μs 0 0 0 254.91 KB
master StringConcatAspectBenchmark net472 289μs 6.53μs 62.3μs 0 0 0 278.53 KB
#6706 StringConcatBenchmark net6.0 57.2μs 620ns 5.72μs 0 0 0 43.44 KB
#6706 StringConcatBenchmark netcoreapp3.1 65.6μs 919ns 9.19μs 0 0 0 42.64 KB
#6706 StringConcatBenchmark net472 37.6μs 137ns 513ns 0 0 0 59.83 KB
#6706 StringConcatAspectBenchmark net6.0 309μs 1.66μs 10.1μs 0 0 0 254.77 KB
#6706 StringConcatAspectBenchmark netcoreapp3.1 305μs 5.8μs 57.1μs 0 0 0 252.89 KB
#6706 StringConcatAspectBenchmark net472 288μs 6.41μs 62.1μs 0 0 0 278.53 KB

@daniel-romano-DD daniel-romano-DD force-pushed the dani/iast/ldap_flakiness branch 2 times, most recently from 6e70cf6 to fcdccc8 Compare February 25, 2025 12:53
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Feb 25, 2025 •
edited
Loading

Datadog Report

Branch report: dani/iast/ldap_flakiness
Commit report: 1399f56
Test service: dd-trace-dotnet

✅ 0 Failed, 247651 Passed, 2371 Skipped, 20h 4m 27.09s Total Time

@daniel-romano-DD daniel-romano-DD marked this pull request as ready for review February 26, 2025 10:06
@daniel-romano-DD daniel-romano-DD requested review from a team as code owners February 26, 2025 10:06
@daniel-romano-DD
Flakiness detection
c44eba8
@daniel-romano-DD
Fix ldap tests
865a4e4 
Snapshots fix

Cover server not ready exception

Fix server not ready exception
@daniel-romano-DD
Fix snapshot hash
1399f56 
Fixed hash and method variability

Fix COM exception catch
e-n-0
e-n-0 approved these changes Feb 27, 2025
View reviewed changes
Copy link
Member

@e-n-0 e-n-0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 😄

@andrewlock andrewlock merged commit 1f9079a into master Mar 11, 2025
144 of 147 checks passed
@andrewlock andrewlock deleted the dani/iast/ldap_flakiness branch March 11, 2025 15:42
@github-actions github-actions bot added this to the vNext-v3 milestone Mar 11, 2025
ddyurchenko pushed a commit that referenced this pull request Mar 31, 2025
@daniel-romano-DD @ddyurchenko
[IAST] LDAP flakiness fix (#6706)
5fb810b 
## Summary of changes
Handle server behavior when it's not ready to handle the LDAP request.
Slip test in this case

## Reason for change
Sometimes LDAP requests crash in CI, unrelated to our instrumentation

## Implementation details
Moved LDAP requesto to a task and handle the timeout and possible
`COMException` thrown

## Test coverage

## Other details
<!-- Fixes #{issue} -->

<!-- ⚠️ Note: where possible, please obtain 2 approvals prior to
merging. Unless CODEOWNERS specifies otherwise, for external teams it is
typically best to have one review from a team member, and one review
from apm-dotnet. Trivial changes do not require 2 reviews. -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@e-n-0 e-n-0 e-n-0 approved these changes

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:asm-iast area:tests unit tests, integration tests
Projects
None yet
Milestone
3.13.0
Development

Successfully merging this pull request may close these issues.
3 participants
@daniel-romano-DD @andrewlock @e-n-0