Fix benchmark failures and improve results quality #8962
Merged
+242
−120
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 9 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.035 s) : 0, 1034811
Total [baseline] (8.523 s) : 0, 8522989
Agent [candidate] (1.027 s) : 0, 1026584
Total [candidate] (8.519 s) : 0, 8518785
section iast
Agent [baseline] (1.164 s) : 0, 1164483
Total [baseline] (9.235 s) : 0, 9234670
Agent [candidate] (1.155 s) : 0, 1154833
Total [candidate] (9.233 s) : 0, 9233273
gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (689.135 ms) : 0, 689135
BytebuddyAgent [candidate] (683.578 ms) : 0, 683578
GlobalTracer [baseline] (242.508 ms) : 0, 242508
GlobalTracer [candidate] (240.983 ms) : 0, 240983
AppSec [baseline] (59.896 ms) : 0, 59896
AppSec [candidate] (58.971 ms) : 0, 58971
Debugger [baseline] (6.137 ms) : 0, 6137
Debugger [candidate] (6.164 ms) : 0, 6164
Remote Config [baseline] (731.997 µs) : 0, 732
Remote Config [candidate] (744.127 µs) : 0, 744
Telemetry [baseline] (12.744 ms) : 0, 12744
Telemetry [candidate] (12.802 ms) : 0, 12802
section iast
BytebuddyAgent [baseline] (811.085 ms) : 0, 811085
BytebuddyAgent [candidate] (804.189 ms) : 0, 804189
GlobalTracer [baseline] (233.157 ms) : 0, 233157
GlobalTracer [candidate] (231.017 ms) : 0, 231017
AppSec [baseline] (54.917 ms) : 0, 54917
AppSec [candidate] (53.996 ms) : 0, 53996
Debugger [baseline] (5.99 ms) : 0, 5990
Debugger [candidate] (5.959 ms) : 0, 5959
Remote Config [baseline] (590.799 µs) : 0, 591
Remote Config [candidate] (588.825 µs) : 0, 589
Telemetry [baseline] (7.921 ms) : 0, 7921
Telemetry [candidate] (7.974 ms) : 0, 7974
IAST [baseline] (27.223 ms) : 0, 27223
IAST [candidate] (27.664 ms) : 0, 27664
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1036094
Total [baseline] (10.721 s) : 0, 10721449
Agent [candidate] (1.03 s) : 0, 1030285
Total [candidate] (10.614 s) : 0, 10614410
section appsec
Agent [baseline] (1.185 s) : 0, 1184885
Total [baseline] (10.786 s) : 0, 10785759
Agent [candidate] (1.179 s) : 0, 1179370
Total [candidate] (10.787 s) : 0, 10787458
section iast
Agent [baseline] (1.153 s) : 0, 1152693
Total [baseline] (11.022 s) : 0, 11022412
Agent [candidate] (1.155 s) : 0, 1154860
Total [candidate] (10.872 s) : 0, 10871626
section profiling
Agent [baseline] (1.279 s) : 0, 1278925
Total [baseline] (10.95 s) : 0, 10950341
Agent [candidate] (1.273 s) : 0, 1273092
Total [candidate] (10.943 s) : 0, 10942677
gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (689.874 ms) : 0, 689874
BytebuddyAgent [candidate] (685.219 ms) : 0, 685219
GlobalTracer [baseline] (242.746 ms) : 0, 242746
GlobalTracer [candidate] (241.816 ms) : 0, 241816
AppSec [baseline] (62.193 ms) : 0, 62193
AppSec [candidate] (60.712 ms) : 0, 60712
Debugger [baseline] (6.138 ms) : 0, 6138
Debugger [candidate] (6.922 ms) : 0, 6922
Remote Config [baseline] (741.16 µs) : 0, 741
Remote Config [candidate] (736.445 µs) : 0, 736
Telemetry [baseline] (10.593 ms) : 0, 10593
Telemetry [candidate] (11.379 ms) : 0, 11379
section appsec
BytebuddyAgent [baseline] (709.252 ms) : 0, 709252
BytebuddyAgent [candidate] (706.81 ms) : 0, 706810
GlobalTracer [baseline] (236.753 ms) : 0, 236753
GlobalTracer [candidate] (234.917 ms) : 0, 234917
AppSec [baseline] (179.594 ms) : 0, 179594
AppSec [candidate] (178.838 ms) : 0, 178838
Debugger [baseline] (5.885 ms) : 0, 5885
Debugger [candidate] (5.841 ms) : 0, 5841
Remote Config [baseline] (619.212 µs) : 0, 619
Remote Config [candidate] (602.423 µs) : 0, 602
Telemetry [baseline] (7.302 ms) : 0, 7302
Telemetry [candidate] (7.202 ms) : 0, 7202
IAST [baseline] (21.997 ms) : 0, 21997
IAST [candidate] (21.703 ms) : 0, 21703
section iast
BytebuddyAgent [baseline] (802.113 ms) : 0, 802113
BytebuddyAgent [candidate] (804.133 ms) : 0, 804133
GlobalTracer [baseline] (230.697 ms) : 0, 230697
GlobalTracer [candidate] (231.056 ms) : 0, 231056
AppSec [baseline] (56.685 ms) : 0, 56685
AppSec [candidate] (55.642 ms) : 0, 55642
Debugger [baseline] (6.013 ms) : 0, 6013
Debugger [candidate] (5.949 ms) : 0, 5949
Remote Config [baseline] (593.947 µs) : 0, 594
Remote Config [candidate] (605.526 µs) : 0, 606
Telemetry [baseline] (7.9 ms) : 0, 7900
Telemetry [candidate] (7.842 ms) : 0, 7842
IAST [baseline] (25.222 ms) : 0, 25222
IAST [candidate] (26.142 ms) : 0, 26142
section profiling
BytebuddyAgent [baseline] (679.578 ms) : 0, 679578
BytebuddyAgent [candidate] (676.185 ms) : 0, 676185
GlobalTracer [baseline] (361.871 ms) : 0, 361871
GlobalTracer [candidate] (360.225 ms) : 0, 360225
AppSec [baseline] (64.314 ms) : 0, 64314
AppSec [candidate] (64.474 ms) : 0, 64474
Debugger [baseline] (6.197 ms) : 0, 6197
Debugger [candidate] (6.162 ms) : 0, 6162
Remote Config [baseline] (667.56 µs) : 0, 668
Remote Config [candidate] (660.839 µs) : 0, 661
Telemetry [baseline] (8.223 ms) : 0, 8223
Telemetry [candidate] (8.155 ms) : 0, 8155
ProfilingAgent [baseline] (107.019 ms) : 0, 107019
ProfilingAgent [candidate] (106.372 ms) : 0, 106372
Profiling [baseline] (107.044 ms) : 0, 107044
Profiling [candidate] (106.396 ms) : 0, 106396
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 1 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section baseline
no_agent (36.997 ms) : 36698, 37295
. : milestone, 36997,
appsec (47.718 ms) : 47299, 48137
. : milestone, 47718,
code_origins (44.506 ms) : 44134, 44878
. : milestone, 44506,
iast (43.875 ms) : 43505, 44245
. : milestone, 43875,
profiling (49.337 ms) : 48857, 49818
. : milestone, 49337,
tracing (44.731 ms) : 44368, 45094
. : milestone, 44731,
section candidate
no_agent (36.876 ms) : 36573, 37178
. : milestone, 36876,
appsec (45.978 ms) : 45579, 46377
. : milestone, 45978,
code_origins (44.696 ms) : 44336, 45055
. : milestone, 44696,
iast (44.73 ms) : 44328, 45131
. : milestone, 44730,
profiling (48.378 ms) : 47931, 48825
. : milestone, 48378,
tracing (44.762 ms) : 44392, 45132
. : milestone, 44762,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section baseline
no_agent (4.411 ms) : 4357, 4466
. : milestone, 4411,
iast (9.283 ms) : 9133, 9432
. : milestone, 9283,
iast_FULL (13.772 ms) : 13500, 14044
. : milestone, 13772,
iast_GLOBAL (10.219 ms) : 10036, 10402
. : milestone, 10219,
profiling (9.022 ms) : 8882, 9163
. : milestone, 9022,
tracing (7.626 ms) : 7503, 7748
. : milestone, 7626,
section candidate
no_agent (4.42 ms) : 4368, 4473
. : milestone, 4420,
iast (9.076 ms) : 8928, 9224
. : milestone, 9076,
iast_FULL (14.507 ms) : 14218, 14796
. : milestone, 14507,
iast_GLOBAL (10.047 ms) : 9861, 10234
. : milestone, 10047,
profiling (8.928 ms) : 8789, 9067
. : milestone, 8928,
tracing (7.755 ms) : 7640, 7870
. : milestone, 7755,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section baseline
no_agent (1.473 ms) : 1461, 1484
. : milestone, 1473,
appsec (2.415 ms) : 2366, 2464
. : milestone, 2415,
iast (2.185 ms) : 2124, 2247
. : milestone, 2185,
iast_GLOBAL (2.228 ms) : 2166, 2290
. : milestone, 2228,
profiling (2.036 ms) : 1987, 2085
. : milestone, 2036,
tracing (2.004 ms) : 1956, 2051
. : milestone, 2004,
section candidate
no_agent (1.479 ms) : 1467, 1490
. : milestone, 1479,
appsec (2.406 ms) : 2357, 2455
. : milestone, 2406,
iast (2.194 ms) : 2132, 2256
. : milestone, 2194,
iast_GLOBAL (2.231 ms) : 2169, 2293
. : milestone, 2231,
profiling (2.03 ms) : 1981, 2079
. : milestone, 2030,
tracing (2.007 ms) : 1959, 2055
. : milestone, 2007,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~e3de5081c7, baseline=1.50.0-SNAPSHOT~72f45a8563
dateFormat X
axisFormat %s
section baseline
no_agent (15.091 s) : 15091000, 15091000
. : milestone, 15091000,
appsec (14.877 s) : 14877000, 14877000
. : milestone, 14877000,
iast (19.129 s) : 19129000, 19129000
. : milestone, 19129000,
iast_GLOBAL (18.039 s) : 18039000, 18039000
. : milestone, 18039000,
profiling (15.176 s) : 15176000, 15176000
. : milestone, 15176000,
tracing (14.976 s) : 14976000, 14976000
. : milestone, 14976000,
section candidate
no_agent (14.968 s) : 14968000, 14968000
. : milestone, 14968000,
appsec (14.79 s) : 14790000, 14790000
. : milestone, 14790000,
iast (18.963 s) : 18963000, 18963000
. : milestone, 18963000,
iast_GLOBAL (17.861 s) : 17861000, 17861000
. : milestone, 17861000,
profiling (15.106 s) : 15106000, 15106000
. : milestone, 15106000,
tracing (14.935 s) : 14935000, 14935000
. : milestone, 14935000,
|
ddyurchenko
changed the title
ddyurchenko
requested review from
AlexeyKuznetsov-DD and
pawelchcki
and removed request for
a team
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
ddyurchenko
added
the
tag: no release notes
smola
approved these changes
Jun 13, 2025
manuel-alvarez-alvarez
approved these changes
Jun 13, 2025
|
LGTM thanks for the update! |
bric3
approved these changes
Jun 13, 2025
Comment on lines
4
to
5
| "service": "bash -c \"${CPU_AFFINITY_K6}${UTILS_DIR}/run-k6-load-test.sh http://localhost:8080 ${OUTPUT_DIR}/${VARIANT} 'pkill java'\"", | ||
| "run": "bash -c \"${CPU_AFFINITY_APP}java ${JAVA_OPTS} -Xms2G -Xmx2G -jar ${PETCLINIC} &> ${OUTPUT_DIR}/${VARIANT}/petclinic.log\"", |
There was a problem hiding this comment.
nitpick: Wouldn't it be better to have a space there after the variable. i.e. to avoid assumptions this new env variable will have a trailing space. This shouldn't be an issue for bash -c
ddyurchenko
force-pushed
the
ddyurchenko/fix-load-tests
branch
3 times, most recently
from
121d51e to
b7d0061
Compare
jpbempel
reviewed
Jun 16, 2025
benchmark/load/run.sh
Outdated
Comment on lines
6
to
7
| export CPU_AFFINITY_K6="taskset -c 25-26 " | ||
| export CPU_AFFINITY_APP="taskset -c 27-47 " |
There was a problem hiding this comment.
Are you sure that the CPU topology on the machine is always the same?
There was a problem hiding this comment.
Yes, this is guaranteed. We explicitly allocate for metal runners cores 0-23 (CPU socket 1) to system tasks and cores 24-47 (CPU socket 2) for benchmarks.
jpbempel
approved these changes
Jun 18, 2025
This reverts commit a8c774e
ddyurchenko
force-pushed
the
ddyurchenko/fix-load-tests
branch
from
e7a237b to
e3de508
Compare
sarahchen6
pushed a commit
that referenced
this pull request
Jun 18, 2025
Fix benchmark failures and improve results quality
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
After updating JVM from 17.0.11 to 17.0.15 in benchmarking Docker image, we experienced slowdown of performance in both macro- and microbenchmarks. As result, some of benchmark runs started failing on timeouts or creep CI performance.
This PR fixes the issue with insecure bank and petclinic apps, and also addresses data quality issue with warmup time, since before we included it into results comparison and skewed results.
Motivation
Fix benchmark runs, make sure that we have results stable and reported again.
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]